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I. INTRODUCTION 

A. BACKGROUND 

This thesis analyzes Department of Defense (DoD) Business 
Process Reengineering initiatives that focus on increasing the 
use of parametric cost estimating techniques in DoD 
applications. Using parametric pricing techniques to estimate 
program costs has a relatively short history in comparison 
with other pricing techniques. To date, cost estimates based 
on parametric estimating methods have been restricted mainly 
to overall initial planning but they have wider applications 
that are now being contemplated through direct policy action. 
It has been proposed that they be applied to price estimates 
for individual contract actions or families of contract 
actions to reduce a potential contractor’s bid and proposal 
costs, proposal cycle time, and to expedite the Federal 
Government proposal evaluation and contract award process. 
B. SCOPE AND ASSUMPTIONS 

The objective of this thesis is to study the policy 
effects of increasing the use of parametric cost estimation 
methods in individual contract actions. Any reference to 
program management uses of parametric cost estimating methods 
is done to relate how individual contract actions fit within 
the framework of a generic program. Anticipated uses in 


specific programs or contract actions are not reviewed or 


analyzed. No attempt is made to validate any existing 
parametric cost estimating models, techniques, processes, 
software, or their applications in specific situations. 

This thesis identifies potential impediments to the 
implementation of the policy and makes recommendations on how 
they can be overcome. Specifically, the affects on proposal 
generation, evaluation, and subsequent contract performance is 
evaluated. The basic effort is directed at producing an 
accurate picture of how parametric cost estimating techniques 
may be used by the Department of Defense (DoD) and contractors 
throughout the contracting process until completion. Many 
studies have been done on the use of parametric cost estimate 
development for entire classes of equipments, individual 
contract actions, or programs, but none have focused on the 
affects of the policy behind the use of the methods. 

It was assumed at the onset of the research that few 
organizations and personnel employed by DoD either directly or 
through a contract vehicle were trained in the use of 
parametric cost estimating methods. It was further assumed 
that of those organizations and people trained in their use, 
most would be concentrated at the program office level or 
higher in buying organizations or in the DoD policy making 
areas. Finally, it was assumed that neither the Federal 


Government as a whole nor DoD as a subset had promulgated 


specific guidance, policy, @e6r iamstrmwWetions regarding the wise 
and application of parametric cost estimating techniques. 
C RESEARCH OBJECTIVE 

This effort is intended to review the increased attention 
being paid to the area of parametric methods in the area of 
cost estimation in contracting. This effort will give greater 
insight into the potential uses for parametric cost estimating 
techniques. An analysis of the problems that may be 
encountered at the onset of policy implementation may lead to 
a concerted effort to meet them with corrective action before 
they become impediments in program management. The analysis 
provides information regarding how companies contracting with 
the Federal Government and Federal Government personnel 
interacting with these companies can effectively use 
parametric cost estimating techniques to support cost 
proposals submitted to the DoD. 
|B RESEARCH QUESTIONS 

Given the stated objectives for this thesis, the areas of 


research are: 


Primary research question: 
How could a DoD emphasis on using parametric methods in 


cost estimating affect the procurement process? 


Subsidiary research questions include: 
1. Whats changesmcould offerors make to» their proposal 
preparation, submission, and support processes’ to 
accommodate using parametric methods, techniques, or 
software in estimatimg potential contract costs? 
2. What impacts could emphasis on using cost estimates 
based on parametric estimating techniques have on source 
selection criteria and on the evaluation of proposals by 
the DOD? 
3. How could source selection criteria and evaluation 
factors be constructed to fairly evaluate a proposal that 
uses a parametric cost estimate with one that does not? 
4. What effects could using parametric cost estimations 
have on negotiations? 
E. RESEARCH METHODOLOGY 
The basis of the study is a literature review through the 
Defense Logistics Studies Information Exchange (DLSIE), the 
Defense Technical Information Center (DTIC), and various 
publications and journals. Data were also collected through 
survey questionnaires from Federal Government policy offices, 
acquisition management organizations, contracting offices, 
companies doing business with the Federal Government, and 
several academic institutions. The survey used is provided as 


Appendix C and the data from the survey are presented in 


Appendix E. Personnel interviews were conducted with several 
representatives from DoD organizations, industry 
organizations, and companies doing business with the DoD. 
Interview questions are presented in Appendix D and the 
information from the interviews is incorporated in the body of 
the text. 
ive ORGANIZATION OF THE STUDY 

Chapter D of this thesis is an introduction providing a 
brief description of the topic, the scope of the research 
effort, and the research methodology used. A review of the 
background of parametric cost estimating is discussed in 
Chapter II. Cost estimation technigues and their historical 
uses in Department of Defense contracting are also discussed. 
Chapter III describes the process of developing a parametric 
cost estimate. In Chapter IV, the data collection methodology 
is described, the data displayed, and data analysis presented. 
Chapter V concludes the research effort with a summary of the 
findings, conclusions, and recommendations. Appendices will 
present definitions and acronyms used and the survey vehicle 


used to collect the data. 








II. BACKGROUND REVIEW AND HISTORICAL PERSPECTIVE 


A. PURPOSE 

The purpose of this chapter 1s to lay the historical 
foundation from which the study was conducted. Beginning with 
the development of parametric cost estimating techniques, the 
chapter next discusses their application in the source 
selection segment of the Federal Government’s acquisition 
mEoOcess . Parametric cost estimating techniques are defined, 
as well as several other terms relating to the acquisition 
process, as they relate to the use of parametric cost 
estimating methods or techniques. Potential applications in 
both Federal Government acquisition and commercial arenas are 
examined. Finally, the chapter closes by discussing the 
advantages and limitations of using parametric cost estimating 
techniques. This information provides the background for 
understanding why there iS an interest in applying the 
techniques more widely than they have been in the past. 
B. BACKGROUND 

1 ae History 

Historically, the use of parametric cost estimates for 
pricing purposes is relatively new. Whereas price analysis 
ioe Oracticed in some form for every purchase that has been 


made, the techniques used in parametric cost estimating have 


their origins in the learning curve theory espoused by T.P. 
Wright for use in the aircraft industry in the latter part of 
the 19s0s— (Ret —3/:p.4} The common sense notion that 
production of an item becomes more efficient as the quantity 
produced increased was finally quantified. Having this 
relationship mathematically derived rather than estimated 
through experience and guesswork, allowed manufacturers to 
begin forecasting requirements more accurately over several 
production runs. Managers could predict the cost of resources 
needed to produce or the delivery schedule for a given item at 
a given time in the production cycle through the use of a 
logarBthmice @Grachmort unmis Of Bereduction and labor hours 
needed to produce a given unit of production. [Ref 37:p.4] 
Following World War II, other estimating techniques were 
developed and refined. Wessel, in his article, describes 
several methods in use at that time to rapidly estimate 
project costs for use in comparative analysis of capital 
investment opportunities. [Ref 46:p.169] Of interest to the 
development of parametric cost estimating is the discussion of 
GOS E-De EM env lo® GoTocuce in the U.S. chemical industmya 
Although this is certainly not the first time ratio analysis 
was used in industry, it is an indicator that the technique 
was in common use. Attempts were also being made at that time 


to increase the accuracy of the estimates through analyzing 


the data and correlating costs to several production factors. 
At that time, the estimates were said to be accurate enough 
to produce results which allowed managers to narrow the choice 
of new products or processes to the most competitive 
available. 
Descriptions of several common estimating techniques used 
by commercial industry in the late 1950s are: 
1. Detailed Estimates are individual cost elements 
compiled from in-depth engineering projections. 
2. Preliminary-Bill Estimates are materials priced from 
price lists or quotations and labor costs derived from 
labor-material ratios seen on similar jobs in the past. 
3. Layout Estimates are equipment costs obtained from 
vendor quotations or price lists and then corrected for 
various inflation factors uSing appropriate indices. 
Material costs are estimated based on some cost per unit 
of output basis, with engineering costs calculated from 
a percentage curve based on past experience with similar 


projects. 


4. Equipment-Ratio Estimates are estimated by using 
ratios developed from experience on past projects. 


5. Capacity-Cost Estimates are estimates made by 
projecting costs for a new plant by using a logarithmic 
plot of similar plant capacities and costs from 
histormeal pmecondsme [Ref 14:p.131]) 
These techniques show increasing interest in producing 
accurate cost estimates and from that interest greater 
sophistication in generating those estimates. While there 


were elements of these techniques that are necessary for 


parametric cost estimating, they had not yet come together. 


More importantly, they did lay the foundation for parametric 
cost estimating. To produce estimates using many of these 
methods the way data were collected, analyzed, displayed, and 
stored had to be changed. 

Concurrent with the interest in the U.S. commercial 
industry with greater estimate accuracy, the Federal 
Government, particularly the DoD, was under pressure to 
develop more accurate estimates. As in industry, the Rand 
Corporation noticed relationships between costs and production 
outputs. [Ref 37:p.7] Whereas industrial applications related 
costs to production quantities or consumption of raw materials 
used, the Rand Corporation analyzed the correlations between 
costs and variables that described operating characteristics 
of amserake. In particular, it was observed that certain 
classes of aircraft had similar cost behaviors regarding 
weight, speed, range, and altitude. These behaviors were 
called Cost Estimating Relationships (CER) and are the basis 
for parametric cost estimating. The combination of learning 
curve theory, available data from extensive production of 
Similar items using similar processes (aircraft production 
during and after World War II), and development of the idea of 
the CER came together in the form of parametric cost 


estimating. 
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The parametric estimation methods have been used for many 
years with ever widening applications being found for their 
use. From some of the original uses in the Department of the 
Navy (DON) as first order estimates relating aircraft and ship 
weight and cost to develop simple and fairly reliable cost 
estimates, the methods have been widely applied. Parametric 
cost estimating allowed planners to make accurate estimates of 
system costs at early stages in design, a solution recognized 
in both commercial and Federal Government applications. The 
Secretary of Defense (SECDEF) became convinced of the utility 
of Independent Parametric Cost Estimates (IPCE) in 1971 as a 
tool to control cost growth in major weapon systems. [Ref 
25:p.12] He issued directives to use the techniques in cost 
estimating at the program level to solve the problem of 
accurately estimating Research, Development, Test and 
Evaluation (RDT&E) andes naselies PLOGUecEl One run emeesEse 
Estimation difficulties in the early stages of production at 
that time were exacerbated by the rapidly escalating pace of 
technological development. It was felt that success in the 
early stages of a program’s life would lead to reductions in 
program cost overruns. In conjunction with management 
difficulties caused by cost overruns, there was enormous 
pressure to reduce the DoD budget. To manage these budgetary 


pressures, SECDEF hoped to have each Service perform IPCESsS at 


1] 


each key decision point in the weapons acquisition process to 
increase the accuracy of cost estimates. 

Since the inception of the idea of parametric cost 
estimating and study by the Rand Corporation, there has been 
increasing application in commercial industry and the Federal 
Government for planning purposes. Initial uses of parametric 
cost estimates centered mainly on [Ref 20:p.12] 

1. Identifying possible cost/performance tradeoffs in the 

design effort. 


2. Providing a basis for cost/effectiveness reviews of 
performance specifications. 


3. Providing information useful in the ranking of 
competing alternatives. 


4. Suggesting a need for identifying and considering new 

alternatives. 
The reasonably accurate estimates in the early stages of 
program development and widespread applicability contributed 
greatly to the acceptance of parametric cost estimating as a 
valid method. Increasing competitive forces have encouraged 
their use in commercial applications while high level interest 
and greater public scrutiny have led to wider use in DoD 
areas. 

Figure 1 describes a general relationship between 
procurement cost estimating techniques used and the time line 


throughout a DoD-managed program. For comparison purposes, 
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the cost estimating techniques used before the advent of 
parametric cost estimating are shown along the base of the 
figure where they might have been used at various times during 
the program life. Parametric cost estimates are used in all 
phases of a program life cycle but are most often the basis of 
estimates in the Program Definition and Risk Reduction 
(PD&RR) (Phase I) of a program life cycle. [Ref 26:p.21] 
Parametric cost estimating 1S perceived as the best method of 
generating a detailed estimate from the available data when 
only mission and performance parameters have been described. 
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Figure 1: Program Phases and Types of Estimates 
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PRO Mes es a NCeCDeLTon, | paramethac wwoSte: éstimatingeshas 
experienced an ever expanding base of uses. Aircraft programs 
have long been the beneficiaries of parametric price 
estimating techniques. Weight has been the primary 
independent variable to parametrically estimate costs for 
aircraft systems. Analysis by the Rand Corporation showed 
that another useful variable that shows a good correlation to 
cost is speed. [Ref 20:p.21] Other analyses have defined a 
parametric relationship between cost and wetted area, aspect 
ratio, number of engineering drawings, number of parts, and 
even time in aircraft development and production programs. 
[Ref 20:p.34] Despite all of the analysis, weight is the most 
consistent and accurate parameter to estimate aircraft program 
costs. [Ref 20:p.36] No longer applied solely to Federal 
Government aviation programs, the techniques have been used in 
a wide variety of Federal Government areas. Projects in the 
Federal Government that have used parametric cost estimating 
techniques include Department of Energy construction of dams 
and mining structures. Commercial uses range from 
construction and aircraft manufacturing to computer software 
generation projects. 

The DoN is heavily dependent on the use of parametric 
pricing Ger mayor systems planning. It forms a Major part os 


shipbuilding and aircraft cost estimating both at the 
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aggregate level (program) and equipment level. The techniques 
are used in projections in shipbuilding to estimate costs 
within each element of the Expanded Ship Work Breakdown 
Sime lures Similar to the CERs observed from eulr crate 
production, one of the most useful correlations is Eh eiteeseunea 
Ship’s gross tonnage to cost. [Ref 35:p.23] One of a ship 
class’s most consistent physical aspects that shows a useful 
correlation to cost is that of gross tonnage, making it a most 
useful variable in ship cost estimation. Other CERs that have 
been developed and used successfully for shipbuilding 
estimates are man-hours to gross tonnage and power rating 
ratios to gross tonnage relationships. [Ref 35:p.37] 

The development and use of parametric cost estimating has 
followed that of the development of computing capability. 
Prior to the widespread availability and use of computers, 
data collection and analysis on the scale required for 
parametric cost estimating was prohibitively expensive and 
time consuming. As computer hardware improved so has the use 
of parametric cost estimating. Software has also made 
tremendous improvements that parallel those of hardware. More 
sophisticated processing of data and user friendly programs 
have combined with the hardware improvements to bring 
parametric cost estimating into common use. These 


capabilities allow quantitative analysis of such accumulations 
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of data that, without the computer, parametric cost analysis 
would not be useful in terms of timeliness even if it were 
possible at all in a manual environment. 

Zs Source Selection Process Development 

Parametric cost estimating occupies an important niche in 
the source selection process. For the contractor proposing to 
do business with the Federal Government, estimate accuracy 
could mean the difference between winning and losing the award 
of the contract. A contractor who uses a poor estimate in a 
Firm Fixed-Price (FFP) contract environment will suffer by 
using that estimate. He will either lose money through 
underbidding his costs if he bids lower than his actual costs 
and wins the contract award, or he will lose business if his 
estimate is too high, he bids too high, and he is not awarded 
the contract. For the Federal Government, a reliance on a low 
estimate could lead to quality control problems. The 
contractor who won the contract with a low bid will attempt to 
limit his loss through contractor cost cutting measures. In 
a Cost—-Reimbursement contrac environment, additwvenas 
management resources will be required of the Government to 
control cost overrun situations. The increased costs will 
also bring greater public scrutiny on both the Government and 
the contractor as cost overruns develop. In either case, 


inaccurate cost estimates damage the competitive industrial 
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base and have adverse impacts on the Federal Government 
procurement process. 

Along with the greater computing capacity, the growth of 
parametric cost estimating depended on the data that were 
avallable for analysis. Other Federal Government applications 
depended on the collection of data for analysis. This data 
collection was made available in large part through dependence 
on detailed engineering cost and schedule estimates. The 
Truth in Negotiations Act of 1962 (TINA) codified cost and 
pricing data requirements that were to be submitted in 
contract negotiations with the Federal Government. In meeting 
Eiese requirementsswith detailedwcost esimmatesym commercial 
industry built the databases required for effective parametric 
cost estimating. 

Cost Accounting Standards (CAS) has also had a 
Significant impact on the use of parametric cost estimating 
methods. One of the basic premises of CAS is consistency in 
estimating, accumulating and reporting costs by a company 
proposing to do business with the Federal Government. Any 
parametric model used by a company must be able to segregate 
estimates of costs to match the data collection system used in 
that company. This in turn should match the proposed work 
breakdown structure (WBS) upon which understanding of the 


contractor’s ability to perform work delineated in the 
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Statement of Work (SOW) depends. Failures in this area could 
make Heeb iiGciit ton poameuyInG Oomdganizatbionpesthe Mefence 
Contract Audit Agency (DCAA), or the Defense Contract 
Management Command (DCMC) to evaluate a company’s cost 
proposal. Differences here could also result in CAS 
violations which in turn could lead to adverse effects on 
prior performance ratings. 

Cost estimating plays a major part of any procurement 
decision from influencing the budgetary processes to final 
source selection. The Competition in Contracting Act (CICA) 
of 1984 followed TINA, and both had tremendous impacts on the 
source selection process in Federal Government procurement. 
Where TINA determined data to be submitted as the basis for a 
cost proposal, CICA clearly allowed for negotiated procedures 
in a competitive environment for Federal Government 
procurement when sealed bidding conditions do not apply. [Ref 
41] Beyond the FAR and Defense Federal Acquisition Regulation 
Supplement (DFARS) guidance, executive agencies have published 
additional regulatory constraints that affect the conduct of 
Federal Government procurement actions. General guidance is 
given, however, in the DCAA Contract Audit Manual. [Ref 
37:Appendix C] Parametric cost estimating techniques are not 
specified for use or directed to be used in any of this 


guidance; rather their use has grown as aée=e sensible, 
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businesslike answer to meet the needs of individual programs 
and procurement actions. [Ref 37:p.7,198] 

In acquisition planning, specified in CICA and depicted 
in Figure 1, parametric cost estimating 1s applied to provide 
a manager with useful data upon which he can make an informed 
decision. Source selection planning for a err ae 
procurement is only a part of the overall plan, but it is the 
place where the uses of parametric cost estimating are best 
illustrated. An “individual procuUmementae act lonemusing 
parametric cost estimating can occur in any of the four phases 
shown in Figure 1. Determining that parametric cost 
estimating is useful in the conduct of a procurement action 
may have the most beneficial outcome when used in the planning 
stages. Parametric cost estimating considerations can best 
be integrated into procurement planning to fully exploit their 
potential. 

Use of parametric cost estimating techniques could first 
impact the individual procurement by affecting the structure 
of the solicitation for an item or service. The buyaing 
activity may choose to construct the solicitation to accept, 
reject or encourage the use of parametric cost estimates. 
Requiring the use of parametric cost estimating techniques may 
in some instances restrict competition to those with the 


experience in not only providing the goods or services 


se 


required but also knowledge and experience with the techniques 
specified. Rejecting the use of parametric cost estimating 
techmrques Vas valid merhode, Ofeusuppommingma price proposal 
could remove a competitive advantage for a company in another 
Situation. 

From the solicitation, the companies submitting proposals 
will construct their proposals in a way to present their 
company as the best value for the buying organization. They 
will present the data required by the solicitation in support 
of their proposal. For those companies familiar with 
parametric cost estimating techniques, this could be a direct 
competitive advantage. Conversely, for those who are not 
familiar with them it could be an administrative burden at 
best and a liability in the competitive environment if they 
are required by the buying agency. The buying organization 
should recognize the differences in how parametric cost 
estimates are developed from the traditional bottoms up 
estimates and structure the support data requirements 
accordingly. In either situation, the buying organization 
should restrict the data to be submitted to only those 
reguired to support analysis of the proposal. 

The evaluation criteria for the procurement, expressed in 
the solicitation, should drive the data collection efforts. 


These criteria are required to be explained in the body of the 
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solicitation to potential offerors. Companies proposing to do 
business with the Federal Government will use these criteria 
as considerations in the construction of their proposal. Each 
buying organization may have a different view of the 
interrelationships between the elements in the evaluation 
Criteria. Parametric cost estimating could have impacts not 
only in the area of cost comparisons and cost realism but also 
in the technical proposal involving understanding the 
requirement. In the evaluation of management capability, 
parametric cost estimating could again demonstrate a company’s 
level of understanding of the requirement as well as affect a 
company’s score on past performance. 

The proliferation of data collection systems, management 
systems, and user friendly statistical analysis programs make 
the techniques of parametric cost estimating more available 
than ever before. The techniques have generated an industry 
whose goal is to provide parametric estimating services and 
tools to potential users. The DoD-maintained Constructive 
Cost Model (COCOMO) was developed to estimate software cost 
estimates. One of the more widely used parametric cost 
estimating models produced by commercial industry is the 
Programmed Review of Information for Costing and Evaluation 
See tCE) . [Refigee/2p.219) The PRICE model was initially 


developed by the RCA Corporation in 1974 to provide 
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engineering and manufacturing costs for equipment having a 
wide range of applications. It has undergone several 
enhancements since its inception and is now a family of cost 
estimating models for hardware and software projects. [Ref 
Z1:p.14 59 PRICE, however, is only one of many models 
developed by industry and the Federal Government to meet the 
needs of program and procurement management. 

The development of parametric cost estimating has 
progressed along with increasing budgetary pressures, and 
technical and management improvements. The current state of 
the art in all of these areas has brought renewed interest in 
their use by the upper echelons of DoD. As a step beyond 
their earlier uses, the present drive is focused on reduction 
of contractor costs from onerous Federal Government reporting 
reguirements and an intense desire to reduce equipment 
development cycle times. 

One way to reduce the burden associated with 

providing cost and pricing data is to permit 

greater use of parametric estimating techniques. 

Properly calibrated parametric techniques can 

accurately estimate costs while reducing bid and 

proposal costs and proposal cycle time and 
expediting the Government evaluation process. ([Ref. 

AQ ls| 

To reach the goal set out in the above memorandum by the 


Under Secretary of Defense, the use of parametric pricing will 


have to become more widely accepted throughout the DoD 
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acquisition community as a cost estimation tool rather than 
viewed aS a program management tool for cost estimates. 
eo DEFINITIONS AND CONCEPTS 

His Parametric Cost Estimating 

Parametrrewcost est imatimg@ime olives MeEnegeo Mlect lonmand 
Seqanization Of histori#eal information, analysis thzough 
mathematical techniques, and relating this information to the 
work output that is being estimated. A parametric cost 
estimate can be defined as an eStimate which predicts cost by 
means or explanatory variables such as performance 
characteristics, physical characteristics, and characteristics 
relevant to the development process, as derived from 
experience on logically related systems. [Ref 37:p.172, 
22:p.293}] Combinations of CERs and their relationships are 
used to describe the likely cost behavior of a project. The 
emphasis in parametric cost estimating is to “focus on the 
cost drivers, not the miscellaneous details” [Ref 37:p.11] 
that are common in other cost estimating techniques. 

Zr Parametric Cost Model 

A parametric cost model is simply the group of CERs and 
logical relationships used together to produce cost estimates. 
They can range from common rules of thumb, such as costs per 
gallon, to elaborate computer models. [Ref l7/:p.12] A 


parametric cost model is one derived from the relationships 


23 


DelLwe Meccedwollwes Ii a miStorical database used to provide 
a top down cost estimate. The buyer is attempting to relate 
what he or she wants to buy in units of capability to the cost 
for it. This is a departure from the traditional industrial 
engineering (or bottoms up) approach of describing what a 
buyer is paying for through the use of detailed cost 
accumulation systems. 

Parametric price analysis is used at all stages of a 
major procurement from the initial estimates to developing 
estimates for costs for components later in production. 
Figure 1 is a representation of how parametric pricing is 
typically used in comparison with other cost estimating 
methods. 

oe Cost Estimating Relationships 

The CER is the key component in the development of a 
parametric cost estimating model. CERs are described as: 

...an algorithm relating the cost of an element to 

physical or functional characteristics of that cost 

element or a separate cost element; or relating the 

cost of one cost element to the cost of another 

cost element. [Ref 37:p.165] 

This idea is rooted in the fact that costs in the production 
of an item are related in a quantifiable manner to its 
physical or performance characteristics. An example of this 


principle in the Contract Pricing Resource Guides (CPRG) 
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describes the procedure of developing a CER for building 
houses and relating the costs to such variables as the number 
of bathrooms, available living area, exterior wall surface, or 
roof surface area. [Ref 1:p.D-31] 

CERs can be broken down into two categories, cost—to-cost 
(overhead rates) and cost to non-cost (cost per pound of 
thrust). It is critical that the cost analyst understand the 
logical relationship between cost and any independent variable 
for the model to be used effectively. This understanding 
leads to knowing the limits of the model and potential effects 
of using it with a new database. 

4. Mathematical Techniques Used in CER Development 

Regression analysis and multiple regression analysis are 
two statistical techniques that can be used to relate the cost 
(dependent variable) to a characteristic (independent 
variable). [Ref 23:p.291] The aim of regression analysis, in 
the form of applying linear, curvilinear, logarithmic, or 
exponential techniques to the data, is to derive the equation 
of a curve that predicts how an independent variable will vary 
with respect to a given parameter or dependent variable. In 
the case of procurement, interest has been focused on the cost 
and schedule estimating aspects. When the parametric 


relationship has been described, it can be used to estimate 
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SV Stetececes oy Substitutingydifferent cost, performance,.or 
other characteristics into the equation. 

A second mathematical method that can be used to derive 
a CER is the Least Squares Best Fit (LSBF) method. [Ref 
37 2 pera This algebraic method minimizes the sum of the 
squared deviations of the observed values of a variable and 
calculated values of that variable. In this method, a series 
of calculations involving the mean values of the dependent and 
independent values and the point-slope formula are used to 
arrive at a formula that describes the CER between the two 
variables. 

Apart” £rom “strict” mathematacady derivations of Eke 
relationships between two variables, plotting a series of data 
points and graphically determining the line of “best fit” can 
also be used to derive a CER. [Ref 37:p.41] This method 
requires plotting the data points and simply drawing a line 
through the data points leaving an equal number of points on 
either side of the line. The drawn line must follow the 
general trend of the data points and every data point must be 
considered. Any outlying data points must not be discarded 
but instead investigated to understand why the variation in 
the data occurred. The point-slope algebraic formula for a 


line can be used to derive the CER in this method if a linear 
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relationship is noted. Curvilinear regression is another 
method available to develop the CER. 
2 Statistical Measures of Accuracy 
The statistical measures of the accuracy of an estimate 
based on the two described mathematical methods can be 
described in terms of: 
1. The Standard Error of the Estimate (SSE) - the 
variance associated with the prediction made from 
the estimating equation. [Ref 25:p.13] 
2. The Coefficient of Determination - the closeness 
of fit between two variables or how much one factor 
is affected by another. [Ref 37:p.46] | 
With these measures, a manager has another tool to assess 
the applicability of the CER to a given program or system. 
Using regression analysis, the data are then analyzed for 
relationships. The smaller the SSE and the larger the 
Coefficient of Determination the more accurate the equation 
will estimate the costs that will be incurred. Examination of 
the estimates made by the equations over the data range is a 
critical step in the evaluation. Often there are breaks or 
discontinuities observed in the data. At these points the 
relationship between one independent variable and the 
dependent variable breaks down and another relationship 


between the dependent variable and a second independent 
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variable becomes a more reliable predictor. There may be 
several CERs for a given item or system depending on the level 
of production or range of a given performance characteristic. 

6. Model Validity 

Comparability between the items or services upon which 
the database is founded and those to be estimated is an idea 
even more basic to parametric cost estimating than the CER. 
Hertling defines parametric cost estimating model validation 
as “the process of determining that a model is a reliable 
predictor of costs for the type of system being estimated.” 
[Ref 17:p.13] This is also an issue in the analog method of 
cost estimating. Analog cost estimating methods are simply 
comparisons between past costs observed on similar types of 
systems or equipments and new ones. Although there may be 
some difficulty in determining what are logically related 
systems or programs that can be compared, the techniques can 
be easily applied if such a relationship can be defined 
between an existing system or program and a projected one. As 
in the analog methods of cost estimating, the greater the 
difference between the programs, systems, or equipments that 
make up the database and the projected system, the less useful 
is the estimate produced. 

The validity of the historical data analyzed is one of 


the most important factors in its later applications. [Ref 
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3p. 12) Data input to the model must be verifiable and 
accessible to the cost estimator. Data accuracy is critical 
and has more dimensions than just similarity between the old 
and new programs, systems, or equipments. Parametric cost 
fsoeimates arewledsketise fa imiiieuemrneresaresitecvedatampoints pier 
development of the basis for the CERs or when the methods of 
performing a task change significantly. The comparability 
between a new system and existing ones whose costs make up the 
database used affect the model’s accuracy. Accuracy increases 
when the methods used for production of like items or systems 
are very similar. An example of this is a system that has 


undergone upgrades, with the new item or system incorporating 


technological innovations one increased performance 
characteristics beyond those previously attained. [Ref 
Zep. 293] 

7. Model Validation and Calibration 


One method of validating a parametric cost estimating 
model involves generating estimates with the model for 
Gemparison with actual coSts inewrred. Ref 3tmpp.18, 76) 
The differences must be explained and understood for the model 
to be useful. Without that understanding, the model must be 
modified and analyzed until it does produce reliable and 


accurate estimates. 


Phy, 


Sotipm@aevon OL a medel as basically the relation of a 
model to a particular instance rather than a general set of 
events. This would be a consideration in cases where a 
company uses a standard commercial version of a software 
product or model for estimate generation rather than a company 
that developed a model based on their own historical data. 
[Reise "- pp.) ame 62) 

8. Work Breakdown Structure 

In the Federal Government acquisition process, the WBS is 
often used as a base upon which the technical requirements for 
a program, system, piece of equipment, or individual contract 
action are developed. [Ref 37:p.177] Through a hierarchical 
family tree structure detailing the end product, work elements 
to achieve production of the end product, and their 
relationships, an understanding of the processes involved in 
the accomplishment of an objective is achieved. Specifically: 

..e-the work breakdown structure plays a significant 

role in planning and assigning management and 

technical responsibilities; and monitoring and 

controlling the progress and status of (a) 

engineering efforts, (b)resource allocations, (c) 

cost estimates, (d) expenditures, and (e) cost and 

technical performance. [Ref 37:p.178] 

Guidance regarding the use of the WBS is provided in MIL-HDBK- 
881. [Ref 37:p.177])@8Deb policy is that MIL=HBBR=@siB shou 


be referenced for guidance only in appropriate solicitations 
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defined as those solicitations involving cost risk to the 
Government. The intent is to improve program and contract WBS 
development, provide guidance to the contractor in extending 
the program WBS and maintain the integrity of the WBS. 

9. Price and Cost Analysis 

Price and cost analysis are used to determine that fair 
and reasonable prices are paid for products in business as 
well as Federal Government procurement. [Ref 2:pp.253, 262] 
Price analysis acts to keep a focus on the value of an item, 
and parametric cost analysis with its focus on cost drivers is 
a clear application of that principle. Price analysis allows 
a buyer to compare a price against a standard as a yardstick 
to determine if an item is worth the cost. Price analysis 
techniques are generally faster and do not cost as much to 
perform as cost analysis, a detailed breakdown of costs for 
each cost element in a proposal, but price analysis cannot 
give an accurate picture of what will be paid for an item in 
all cases. Price analysis is principally performed using 
these four methods [Ref 2:p.253]: 

1. An analysis of competitive price proposals is 

conducted. 


2. A comparison with regulated, catalog, or market prices 
is performed. 


3. A comparison with historical prices is performed. 


4. An independent cost estimate is used. 


31 


Price analysis only provides estimates of those costs 
relative mer tThe™- accuracy of thewprediction: It does not 
provide exact price predictions about costs that will be 
incurred in an effort. 

Parametric cost estimating ties together the concepts of 
price analysis and cost analysis. It is one of the methods of 
comparing a price in the present with those historically paid 
for an item and is one of the four methods a buyer has 
available to perform price analysis. Parametric cost 
estimating combines the price analysis technique of 
referencing historical prices with the mathematical analysis 
inherent in improvement curve theory from cost analysis. [Ref. 
2D 22e 

10. Cost Realism 

Cost realism is a key concept in the source selection 
process entailing understanding a cost proposal in detail and 
evaluating the accuracy of component costs. The purpose of 
cost realism is to give a source selection official greater 
assurance that the effort will cost what is estimated in the 
proposal when the work is complete. [Ref 6:p.3-1] The 
inference is that the more realistic the proposal the more 
likely the project will avoid a cost overrun situation. 


Specific information about how a cost proposal will be 
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evaluated should be included in the source’ selection 
evaluation plan for a given procurement action and published 
in applicable solicitation documents. In general, the Federal 
Government will assess cost realism based on the difference 
between the contract offer and an Independent Government Cost 
Estimate (IGCE). Another method of determining cost realism 
is to compare the proposed costs with a recalculation of the 
costs by the buying command based on contractor-provided data, 
past performance, and information from DCAA or DCMC audits. 
Meet 6:p.11-3] 
De: PARAMETRIC COST ESTIMATING APPLICATIONS IN ACQUISITION 
From the original efforts made by the Rand Corporation to 
develop a method of accurately estimating costs of proposed 
new systems, the uses of parametric cost estimating have 
expanded. Parametric cost estimates have been and continue to 


be used to [Ref 37:p.71] 


1. Identify possible cost-performance tradeoffs 


2. Provide a basis for cost/effectiveness review of 
performance specifications 


3. Provide information useful in ranking competing 
alternatives 


4. Identify and consider new alternatives 
5. Develop target cost estimates for planning purposes 


6. Estimate new program costs 
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7. Make decisions during bid or proposal strategizing 

8. Cross check actual costs during program life cycle 

with projected cost estimates made during planning. 
All of these uses have been primarily in the program 
management area. In the later stages of a program, they are 
used in a secondary role to bottoms up estimating. [In fact, 
the CPRG has the use of CERS in price analysis shown as a 
secondary comparison technique. Emphasis 1S now on using 
parametric cost estimates as a primary source of cost 
estimates on an individual contract action basis. The primary 
goal is to maximize efficiency and minimize complexity of the 
solicitation, the evaluation, and the selection decision 
process. 


Ee ADVANTAGES OF AND LIMITATIONS IN USING PARAMETRIC COST 
ESTIMATING TECHNIQUES 


i Advantages 

Parametric pricing using CERs has the advantages of being 
guick and simple to apply once the relationship has been 
described adequately. [Ref 37:p.38] This is a result of the 
reduced need for cost data specific to the cost elements of 
the new item or _ system. The only data needed are the 
independent variable inputs to the equations to get an output. 
Efforts normally used to generate detailed cost estimates, 


that may be no more accurate than the parametric estimates, 
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can be allocated to other more profitable endeavors. Where a 
detailed estimate may not be possible because of the lack of 
data, a parametric cost estimate may be possible. 

Parametric cost estimating methods do not have to be 
isolated to use only in large scale applications at the 
program or even entire equipment level. [Ref 37:p.68] They 
can be applied at the component level for inclusion in another 
type of cost estimate as well as on the macro management 
level. They provide a relatively quick and accurate way to 
produce cost estimates on a program or system level to be used 
as a sanity check. On the larger scale, when used to estimate 
costs at the system or program level, the data base includes 
costs incurred as a result of setbacks or other problems. 
With the database built on a continuing series of changes, 
usually in high technology areas, the model takes advancing 
technology automatically into account. Thus the CERs have a 
built-in correction factor for problem areas often observed as 
well as for program cost escalations and inflationary 
pressures. 

The accuracy of parametric cost estimates has improved 
Susmiricantly@since thesr wmceptwonmemibRef 25:p.5]) Using 
proven databases parametric cost estimating models are 
outstanding predictors of accurate estimates. From order of 


Magnitude estimates in the early chemical industry 


35 


dpoltcamlons, Eemmocing Caqually as acgurate as detailed 
eng  necmingeeces: sesimauas: in 19/jaa(/Ref 25:p.6], the accuracy 
of parametric cost estimates has improved to the present 
claims of five percent difference from actual observed costs 
in seme uses of the PRECK emodelis. w[Ref 21:p.147] As in any 
modeling situation, accuracy of the inputs is a critical 
factor in obtaining useful and accurate output data. 
Parametric cost estimate accuracy is highly dependent on the 


following variables [Ref 14:p.129]: 


1. Degree of project definition 


2. Comparability between projects in the database and the 
proposed project 


3. Status of model calibration and validation 

4. Data availability and depth of the database used 

5. Adjustments made to the data to ensure comparability 

within the database 
Another factor to consider is the increasing accuracy levels 
achievable throughout the life of a See As more is known 
about a project, more accurate estimates can be made over 
time. The benefit of a parametric cost estimating system is 
the speed at which revised estimates can be generated as new 
information becomes available. The flexibility inherent in 
the parametric cost estimating methods is also demonstrated in 


their uses in conducting “what if” analyses. 
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an Limitations 

Despite all of the advantages, there are some 
difficulties with using parametric cost estimating methods. 
They may not be applicable in all cases because one may not be 
eee tO find a close relationmstie (berry sonmecost ce and wom, 
independent variable that can be easily measured or 
quantified. In some cases the models use CERs that do not 
provide sufficient detail to accurately predict costs. Le 
maintain accuracy, the historical data base must be updated 
continuously, or the equations may not accurately represent 
the relationship between costs and the independent variable 
chosen as a basis for estimation. Technological changes in 
manufacturing processes, to the equipment itself, Or 
Significant changes in the management structure of an 
organization may make it to difficult to compare and correlate 
with the established database. [Ref 14:p.130] 

Additionally, several problems distort the results from 
parametric price estimation. Aside from inaccuracies that can 
be introduced by design into the equations to allow greater 
applicability, the size of the data base can have dramatic 
effects on the accuracy of estimates. [Ref 14:p.132] Larger 
data bases will generally have greater accuracies than smaller 
ones as the statistical models increase in accuracy. Data 


point dispersion will also affect the accuracy but this can be 


37 


measured through the use of SSE and coefficient of 
determination values. The standardization and normalization 
of data that is required to be able to compare the results 
across programs can also adversely affect the output of a 
parametric cost estimating model. 

Two such areas of normalization are in the adjustments 
for inflation by Price Level Adjustments and Cost to Quantity 
Adjustments. [Ref 16:p.120] The data to be analyzed must be 
adjusted to a base year constant dollar level for comparison 
purposes through the use of standard indices. To compare 
across different procurement sizes in terms of quantities of 
units purchased, the data used in analysis must also be 
adjusted to develop consistent cost measures. The most common 
method used to do this is through the use of learning curve 
and average cost per unit calculations and techniques. [Ref 
Sj Byers eS) 

As accurate as parametric cost estimating is, it cannot 
give a 100% accurate picture of what will be paid for an item. 
The nature of estimating is to predict costs in the future 
based on extrapolating known data forward in a new situation. 
The only way to obtain 100% accuracy is to have full 
visibility of costs throughout the life of the project and 


total them at project completion. 
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Development of a parametric cost estimating model and its 
maintenance 1S expensive. Database management itself is 
costly as iS maintaining a system of data collection to build 
and update the information required to keep a parametric cost 
estimating model up to date. Without constant maintenance, 
CERs and the model that is based on them can fall out of date 
rapidly. Finally, although parametric cost estimating models 
are very flexible within the boundaries in which they were 
derived, the degree of difficulty in redefining those 
boundaries makes them inflexible in accommodating major 
program changes. 

F. SUMMARY 

Parametric cost estimating has enjoyed a long history and 
it is used in numerous applications. The foundation of 
parametric estimating system is the integrity of the 
historical database and the CERs that make up the model. 
Before any parametric model is used, the model and database 
must be examined to ensure that they are up to date and 
accurate. The model should be calibrated for a specific 
Situation and then validated before it is relied upon. With 
this information as a base, the following chapter will use the 
concepts addressed in this chapter to develop ae cost 


estimating model using parametric methods. 


a8 








III. DEVELOPING A PARAMETRIC COST ESTIMATING MODEL 


Pox PURPOSE 

This chapter will describe a process by which a 
parametric cost estimating model might be developed. Knowing 
how a model was developed allows the user to understand the 
strengths and limitations of the model. sUlatlLicy Ra loy ebicialem ab iar 
lead the user in the most beneficial application of the model 
to the data available or give him the knowledge he will need 
to optimize the model to meet his specific needs. 

There are eight basic steps in the construction of a 
parametric cost estimating model. [Ref 25:pp.8-35] 

1. Gather data on both costs and potential independent 

variables such as speed, weight, or number of lines of 

computer code. 

2. Plot the data. Through a simple visual analysis, the 

user may determine that there is no useful correlation 


between cost and the independent variable. 


3. Mathematically or graphically relate the data points 
is the precess of CER formulacion. 


4. Examine the CER. Does the CER logically apply to the 
Gata? 


5. Refine the CER and review the data. 
6. Test the CER. Does the CER predict costs comparably 
to known historical costs for the variables used in the 


CER’ 


7. Combine the CERs in the Parametric Cost Estimating 
Model. 
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8. Test the Parametric Cost Estimating Model. Does the 
model predict costs comparably to known historical costs? 


POEnEicmelmeOocCs (Or seis CLrioOtEeim model construction, a study 
by Raymond E. Moore III in 1975, Cost Estimating Relationships 
For Naval Surface Ship Electronic Warfare Equipment, will form 
the basis. [Ref 25] In that study, Moore developed a 
parametric cost estimating model for electronic equipment 
based on a review of the costs and technical data for 12 
existing systems. These data are shown in Table 1. 


Table 1 
Electronic Warfare Equipment Data Matrix 


System Unit Year Developm Instal Weight Volume Sensitiv Power 
Cost ent Cost lation ity Usage 
Cost : 
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Cm a 
Taam [aoe [7 [aeoo [oro [20eo [ase [oes 
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Quantity 








Gain 


















Source: Reference 24, p. 40 
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B. PLOTTING DATA 

Moore compiled these data and analyzed the relationships 
between both development and installation costs and all of the 
parameters listed in Table 1. (Rete. OemekL—?2 | | For 
illustration purposes, this study will demonstrate the second 
step in the development cycle by plotting the system weight 


against the development costs in Figure 2. 


Electronic Equipment 
Weight Vs. Cost 
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Figure 2: Weight/Cost Data Plotted Graphically 


source: Developed by researcher. 


At this point, a visual inspection shows that although 
there are some outlying data points, it appears as if a useful 
mathematical relationship can be derived from the points. In 
contrast, Figure 3 is a graph of system development costs 
plotted against quantities of the systems acquired. From 
visual inspection of the graph there appears to be no useful 


correlation between these two variables. 


43 


& 
Electronic Equipment 
Quantity Vs. Cost 
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Figure 3: Quantity/Cost Plotted Graphically 


Source: Developed by researcher. 


To develop a CER to be used in parametric estimation, the 
buyer has to first decide what he wants to estimate (cost, 
schedule, material, labor, or some other parameter) and then 
determine how the variables are related in the work to be 
performed. Will the price estimate be derived directly from 
the data or tied to a variable to which a cost is then 
applied? Here, it has been determined that cost will be the 
dependent variable to be derived directly from the independent 


variable, weight. 
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oe CER DEVELOPMENT 

Having selected the characteristic that will give an 
accurate representation of the relationship between the 
independent and dependent variables, the relationship must be 
Quantified. The independent and dependent variables must be 
not only related in some manner, but they must be related in 
a definable way. There has to be an understandable 
relationship between the data points, and the data points must 
be easily measurable or inferred from available data. 

Using linear regression technigues a line can be used to 
approximate the relationship between the data points. Here 
the relationship between development cost and weight can be 
described as Y= (.950082 x System Weight) + 604.4864. This 


CER is shown in Figure 4. 


Electronic Equipment 
Weight Vs. Cost 
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Figure 4: Cost/Weight CER Overlaid on Data Graph 


Source: Developed by researcher. 
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The coefficient of determination for this CER is 
Ca] Cuulaik: c Cemem CmG O10 > anche SO ohasto. be 4519.926. A 
coefficient of determination with a value of 1.0 indicates a 
perfect correlation between the two variables. In this case, 
for illustration purposes, the coefficient of determination is 
close enough to a value of 1 to be useful as an indicator of 
future costs. The SSE is relatively large, in part due to the 
wide range of development cost values used in the computation 
and the small database used. 
D. COST ESTIMATING RELATIONSHIP REVIEW/REFINEMENT 

A test of the CER developed here shows that it does make 
intuitive sense. As one would expect, costs appear to rise as 
the weight of the system increases. An examination of the 
relationship and the data supporting it are now made in an 
attempt to increase the accuracy of the CER. Should some of 
the outlying data points be discarded and is the mathematics 
used logically sound are two questions that have to be 
answered each time the CER is used. As an illustration to 
show the effects of data manipulation, two data points will be 
discarded and the results shown graphically in Figure 5. 

Deleting the two data points from the database has a 
great effect on the interpretation of the information. The 
CER developed has changed to Y= .834494 (Weight) + 1300.296 


With a coefficient of determination ome./77771Smend a SSHPoE 
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1643.74. While this action has increased the accuracy of the 
CER, it may have also reduced the effective range over which 
it is valid. Any use of the two CERs developed must take 


these data manipulations into account. 
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Source: Developed by researcher. 


E. COMBINING CERS INTO PARAMETRIC COST ESTIMATING MODELS 
Concluding the data manipulation and CER refinement 
initiates the final step in the process of CER development. 
Mieenis CcomStruction, historical costs are compared with costs 
that would be projected by the CER using the known weights for 
the system. Table 2 presents those data for review. Here, it 


will be assumed that the CER data compares favorably with the 
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historical data and that they fall within an acceptable range 
EGmueOsSt I matingucosts:. 


Table 2 
Actual vs. CER Projected Costs 


System | Weight Development CER Projected 
——— =——— 


SLQ-19 | 5190 188600 5535 


were 
SLQ-30 2060 2562 

ree 
re 


Source: Developed by researcher. 
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Up to this point, the construction has been concerned 
with developing a single CER for estimating development costs. 
The purpose of the effort, however, is to derive a parametric 
cost estimating model for a new electronic system. 
Development costs comprise only a part of the costs of any 
system. Other costs that might be considered are installation 
costs, spare parts, and training for operators. To build a 
parametric cost estimating model, all of these factors must be 
combined in some form. They may be estimated using a CER, as 


were the development costs, or by any of the other cost 
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estimating techniques available. A parametric cost estimating 
model considering these and other factors developed by Moore 
for this data set can be described by the following equation 


(Ref. 26:5. 2ae).; 


PROC = 2811.59351 + .05553 (DEVC) + .02824 (Weight) + 3.81016 
(Volume) + 25.98703 (Sensitivity) + 73.68718 (Power Usage) + 
Pewee o599 (Gain) —- 166/7.51343 (ACY) —- 1027.33765 (LG) — 
is Cres 232 4 wkSM) 


which had the following statistics: 
Coefficient of Determination= .9997 
Standard Error of the Estimate =65.119 


where: 
PROC - Procurement Cost DEVC - Development Cost 
ACT - Active Equipment Code LG - Large Vessel Code 


SM - Small Vessel Code 

With the parametric cost estimating model complete, there 
is only testing against historical costs to verify the 
accuracy of the model remaining. The process described above 
is illustrative of one method available to develop a 
parametric cost estimating model. Using the parametric cost 
estimating model developed, the methods can be used in any of 
a program’s life cycle stages for any number of estimating 


purposes. 
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1 SUMMARY 

This chapter developed a CER using regression analysis to 
demonstrate the relationship between the database used and the 
basis for the estimate. From this and other CERs particular 
to the equipment that is to be developed, a model would be 
constructed. This model would then be calibrated if that 
action was required and finally validated. After that point, 
it could be used to estimate the costs of electronic systems 
of a similar nature to the ones that made up the database. 
The next chapter describes the data collection and analysis of 


the data from surveys and interviews. 
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IV. METHODOLOGY AND DEMOGRAPHICS 


A. PURPOSE 

This chapter will discuss the data collection methods 
weed in the study. Construction of the survey vehicle, 
Appendix C, and interview questionnaire, Appendix D, will be 
described followed by a discussion of the methods used to 
disseminate the survey. Data in this chapter will be 
presented as aggregate information describing the demographic 
breakdown of the respondents. 
BY DATA COLLECTION 

Data collection was centered around the- primary and 
secondary research questions. Information regarding the area 
of interest was collected throughout the use of a self- 
administering survey and personal interviews. The research 
questions formed the backbone of the survey, breaking the 
Survey into discrete areas of interest. The survey was 
structured to follow a possible pattern of use for parametric 
cost estimating techniques through an acquisition cycle. Each 
major question is broken into several subsets to allow greater 
analysis of the topic in question. 

Questions one through four form the basis of the 
discussion of the use of parametric cost estimating in the 


procurement process. Ascertaining the risks inherent in the 
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program, system or equipment, its similarity to existing 
efforts, and the development of the databases available for 
analysis are key elements in determining how parametric cost 
estimating can be used in acquisition planning. Questions one 
through four represent some of the management areas to which 
parametric cost estimating have been and can be applied within 
the acquisition cycle. 

From the basis formed by questions one through four, the 
survey moves on to obtain opinions regarding the subsidiary 
research question: What changes will offerors make to their 
proposal preparation, submission, and support processes to 
accommodate parametric methods in estimating contract costs? 
Question five transitions from the general uses of parametric 
cost estimating methods to issues specific to the conduct of 
pre-award contract actions. Question seven continues to 
gather opinions regarding the question of offeror proposal 
preparation. The buyer’s estimation of his position regarding 
the purchase of an item will be based on his understanding of 
the risks in the acquisition cycle and will be reflected in 
the solicitation he develops. 

The structure of a procurement action is in large part 
affected by its place in the acquisition cycle of a program 
and the overall acquisition strategy. In turn, an offeror’s 


proposal is driven by the solicitation to which it is 
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responding. Elements of question six and question eight 
address the subsidiary research question: What impacts will 
emphasis on parametric cost estimates have on source selection 
criteria and on the evaluation of proposals by the DoD? The 
remaining elements of guestion six and question nine induce 
the survey respondent to consider the manner in which a 
solicitation is conducted. 

A central tenet in the Federal Government acquisition 
process is the fairness of the process. These questions 
collect data relating to the subsidiary research question: How 
will source selection criteria and evaluation factors be 
constructed to fairly evaluate a proposal that uses a 
parametric cost estimate with one that does not? The final 
area addressed in the survey is the negotiation process. 
Question ten collects opinions to answer the question: What 
effects will using parametric cost estimating have on 
negotiations? An additional page was attached for comments 
regarding the questions or any other concerns expressed by the 
respondents. These additional data have been incorporated 
into the analysis where applicable. 

In each of the questions in the survey, the respondents 
are asked to record their opinions about the use of parametric 
cost estimating methods using a numerical scale. Five point 


numerical scales were selected to provide a consistent, simple 
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method of coding the responses for data reduction and 
analysis. Two different scales were used in the survey. 
Questions one through four used a scale describing the utility 
of parametric cost estimating methods and questions five 
through ten used a scale describing agreement with the 
statements that make up the elements of those questions. 
Demographic data were also collected in an attempt to describe 
the population of respondents. The data were examined for 
relationships between subgroups represented in the survey 
respondent population and responses for particular questions. 
Through these methods the data were reviewed, analyzed, and 
related to the research questions. 

An interview questionnaire, Appendix D, was developed in 
the same manner as the survey instrument. The basic and 
subsidiary research questions were addressed in the interview 
questionnaire. The interview questionnaire was presented to 
seven individuals in personal interviews at the Naval 
Postgraduate School over the course of the research. 
Additionally, an initial form of the interview questionnaire 
was given to six people via telephone and E-mail at the start 
of the research effort. The information provided has been 
inserted in the discussion and background areas preceding this 
chapter as well as in the analysis of the survey data where 


applicable. 
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The survey was distributed as widely as possible to 
attempt to include as many elements of the DoD acquisition 
community as possible. The bulk of the surveys were 
distributed by mail using Federal Government agency mailing 
lists and organizational information. Electronic mail and 
facsimile transmissions were also used to distribute 
approximately 25% of the surveys. Potential survey 
participants were also identified through INTERNET searches 
and referrals during interviews. Through the survey’s wide 
distribution across the DoD acquisition community, an 
understanding of the potential for the uses of parametric cost 
estimating methods was generated. Appendix E is.a collection 
of the data derived from the survey effort. 

An important group solicited for input were the members 
of the Federal Government’s Parametric Estimating Initiative 
Steering Committee. This group was identified following the 
preface in Reference 36 and represents a cross section of the 
Dem. acquisition community. The Steering Committee came 
together to identify methods of promoting the uses of 
parametric cost estimating technigues throughout the DoD 
community. For purposes of this survey, they responded to a 
pilot survey with recommendations and suggestions for 
improvement to the survey structure, direction, and the 


wording of questions. Their answers to the questions were 
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also most valuable in that they formed a basis for comparison 
against which the other demographic groups could be compared. 
Resident in this group is the most current information about 
the uses of parametric cost estimating across the entire 
spectrum of actions inherent in the DoD acquisition process. 
The Steering Committee is referred to as the control group 
throughout the analysis portion of this chapter. 
Cx DEMOGRAPHIC DATA 

The survey distribution was tracked through the employer 
question of the survey. The industry and Federal Government 
employer categories were all-encompassing, mutually exclusive 
groupings that were easily tracked. Using this-breakdown as 
a basis, 1260 surveys were distributed over the period 21 
pigust “996 te 15 October 199c. Throughout the research 
effort, it was found that many addresses were incorrect. It 
was assumed that the incorrect addresses resulted from base 
closures and realignments or agency reorganization efforts. 
Where possible, correct addresses were obtained and the survey 
redistributed. Figure 6 represents the survey distribution 


throughout the data collection effort. 
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Survey Distribution 
Employer Groups 
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Figure 6: Research Survey Distribution 


Source: Developed by researcher. 

The majority of the surveys were sent to Federal 
Government contract support activities, DCMC and DCAA (52%). 
It was felt that each organization interacting with a major 
portion of the DoD acquisition community across the entire 
spectrum of procurement actions would represent the DoD 
community experience with the uses of parametric cost 
estimating methods. 

A low survey return rate was observed from the data 
collected from 210 surveys returned. Only) 16 s5s=°oL the 
surveys distributed were returned with data to be input for 
analysis. No opinion or no response to one or more questions 
was recorded on 47% of the surveys returned. Figure 7 shows 


the distribution of the returned surveys. 
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Group Return Rates 
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Figure 7: Survey Return Rates 


Source: Developed By Researcher 


58 


The percentages in the Returned Surveys chart of Figure 
7 display the percent of the surveys returned from an employer 
group. The Group Return Rate chart of Figure 7 shows the 
percentage of the surveys sent to each group that were 
returned by members of those targeted employer groups. The 
percentages of the surveys returned by groups ranged between 
683 from industry participants to a low of 3.64% by Department 
of Energy (DoE) personnel, both small groups within the survey 
population. Despite the relatively small number of the 
industry participants targeted (2%), their high return rate 
made them the fifth largest group contributing to the 
database. The industry group may have had a disproportionate 
affect on the response averages because of this high return 
rate. The remaining groups had fairly similar return rates 
within a much smaller range of 12% to 25% and formed the 
majority of the data reviewed. 

As the survey was designed to ensure anonymity of the 
respondent, no effort was made to track response rates by 
organization within an agency, individually, or by any other 
demographic breakdown other than that shown above. The 
researcher was contacted on an individual basis, however, by 
approximately three percent of the survey population who 
indicated that they would not be participating. All of these 


respondents indicated that they were unfamiliar with the topic 


Be 


and did not feel qualified to join in the survey. While 
parametric cost estimating methods have been in use within DoD 
for many years, their use has typically been isolated to upper 
and middle management areas dealing with major programs. Their 
use has not been as widespread in the other areas of DoD 
procurement that make up the bulk of the procurement 
transactions. [Ref 45:pp.7-11] It is likely that a lack of 
experience with parametric cost estimating methods is a major 
factor in the survey’s low return rate. 

The timing of the survey may also have had a negative 
impact on the return rate observed. The survey was 
distributed at the end of the Federal Government’s fiscal 
year, typically one of the most active times of the year in 
the DoD acquisition community. Although the length of the 
Survey was minimized to encourage participation, the survey 
had 55 individual data elements to be considered. Many of the 
recipients may not have had time to complete the survey within 
the requested time frame and thus chose not to participate. 

Demographic data were requested in four areas beyond the 
employer classification previously discussed. Participants 
were asked to identify themselves by their job classification, 
the length of time that they had been working in this area, 
how they categorized their work, and their level of education 


in mathematics. Each question was examined with these 
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categories in mind to identify trends, correlations, or points 
of interest in the data collected. In terms @6£ Zeb 
classification, three categories comprised the majority of the 
Survey respondents: those identifying themselves as Cost or 
Price Analysts (26%), Contracting Officers (23%), and Auditors 
(22%). Small numbers of participants identified themselves in 
the remaining job classification groups. 

Two employer groups were very homogeneous in composition: 
the DCMC and DCAA participants. The DCMC group identified 
themselves almost entirely within the job classification of 
contracting officer. The DCMC personnel made up 70% of the 
responding individuals of those identifying themselves as 
contracting officers. Unlike the DCMC participants, the DCAA 
group makes up almost all of the auditor job classification 
respondents. The DCAA and Auditor groups are nearly 
synonymous, with 96% of Auditors also classifying themselves 
as working for DCAA. All of the other groups were made up of 
combinations of the job classifications. 

The majority, 49%, of the participants identified 
themselves within the acquisition community and only 19% in 
the cost estimating community. A large portion of the 
respondents did not chose to respond to this question, perhaps 
@verlooking it. There was no observed correlation between 


either community and any other demographic category similar to 
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that observed @m the auditor and contracting officer job 
Classifications. The majority of the respondents were very 
experienced in their positions, having more than 10 years of 
experience in either the acquisition or cost estimating 
professions. Very few people, 11% of the 210 participants, 
had less than 10 years experience. Also of note, only 39% of 
the respondents chose to provide their GS/SES level or 
military rank. The resulting analyses from these groups are 
limited because of their small size. 

The self-described work classifications are unlike the 
other demographic breakdowns in that these groups are not 
mutually exclusive. In question four, respondents were asked 
to select any of the work functions listed that they 
performed. They were not to limit their response to any 
Single work function that made up the majority of their work 
effort. Reviewing or auditing estimates was chosen to 
describe the work most often performed by the respondents; 103 
selected this work description. This group’s answers roughly 
correspond to those in the job classifications of cost and 
price analyst and auditor. Negotiation preparation and 
proposal analysis was also selected by a large number of 
participants, 20L, of) 210,5to describe their work efforess 


Here, there is a correlation between this group of responses 
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and those of the contracting officer, contract negotiator, and 
auditor job classifications. 

Within the database, the majority, 883, of the 
respondents answered that they had some degree of college 
mathematics education. A significant number of the 
participants had either college level algebra or business 
calculus, 30% and 33% of the respondents respectively. Oi 
those identifying themselves as having taken an engineering 
calculus series, only 30% also identified themselves as 
engineers. The remaining individuals identified were spread 
across the rest of the job classifications as were the people 
who answered that they had either advanced degrees or majored 
or minored in mathematics while a college undergraduate. 

After reviewing the basic statistics for the entire 
database, the database was subdivided and analyzed. The 
information presented above and the demographic description in 
Appendix E were used to break the database into separate 
components to be compared against each other and the entire 
database statistics. In the following Chapter the comparisons 


made and analyses performed on the data will be described. 
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V. APPLICATIONS IN PROGRAM MANAGEMENT FUNCTIONS 


A. PURPOSE 

This chapter and the following two chapters will discuss 
the data analysis methods used and results observed in the 
study. Chapter V analyzes results from question one data 
related to parametric cost estimating applications in the 
broad scope of program management functions. Data pertinent 
to survey question one will be presented, related to the 
research guestion, and analyzed. Questions one through four 
used a five point scale that ranged from “vital to the 
procurement process,” a one on the scale, to “a-hindrance to 


ot 


the procurement process,” a five on the scale. Each question 
was further broken into smaller areas of specific interest. 

The data from each question were analyzed and the 
following statistics derived: 

1. The number of responses in each of the five 

categories. 

2. An average value of the responses. 

3. The standard deviation for the responses. 

4. The variation of the responses. 


5. The median value of the responses. 


6. The skewness of the responses. 
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These data are presented in Appendix E for the entire group of 
responses as well as for each demographic group. 

Analysis of all the questions and their components was 
conducted in a Similar manner. Descriptive statistics for the 
entire group are presented, followed by group breakdown 
information, and the data presentation closes with analysis of 
varlance and regression information. These data are presented 
in tables following an introductory paragraph. Group data for 
each question was chosen to be displayed for one of three 
reasons: 

1. Some degree of consensus was observed in the group 

data analysis 


2. The group appeared aS a point of interest in the 
analysis of variance, or 


3. The group data are illustrative of a point of interest 

found during the regression analysis performed. 
In both the regression analyses and analyses of variance, a 
95% confidence interval was used. For simplicity, groue 
descriptive statistics for the first subgroup in the first 
question, describing the utility of parametric cost estimating 
in the area of benchmarking process costs, will be presented 
in tabular and text format with the data for the remaining 
questions shown in tabular format only. An analysis of the 


displayed data will follow those data for each question. 
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B. BROAD SCOPE UTILITY 

The first question asked the respondent to provide an 
opinion regarding the utility of parametric cost estimating 
methods in several functional areas. This question was 
divided into 12 segments, five dealing with. overarching 
functions common to the acquisition cycle in general and seven 
treating procurement of specific items or services. 

i). Benchmarking Process Costs 

Benchmarking process cost was the initial subgroup within 
eiesction 1. Parametric cost estimating methods may have 
applications in situations where new processes are being 
developed. The methods may be most useful-where these 
processes are sufficiently similar to past or existing 
processes and the new processes are expected to have similar 
cost characteristics or patterns. AS a group, the 
respondents’ opinions indicated that they thought parametric 
cost estimating methods were somewhat useful in this 
PieetOonal area. The largest™single group, 56 Hespondents, 
selected somewhat useful, a three on the scale, as describing 
the methods’ degree of utility here. Only one respondent 
thought that the methods could hinder cost estimating if used 
here and 21 people thought they were of no use. 

The corporate policy group selected either vital or very 


useful to describe the utility of parametric cost estimating 
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MethOdsenerhimemsitivalion. This agreementmiisestatistuieally 
demonstrated in the small standard deviation of .548 for the 
group and a mean value of 1.4 for all of the responses. These 
statistics must be tempered, however, with the group size. 
With only five individuals, it is a very small part of the 
database of 210. Other small groups had similar occurrences 
in the data analysis. The program staff group, five persons, 
three industry cost analysts and the U.S. Army cost analyst 
group, a group of nine, were unanimous in their belief that 
parametric cost estimating methods were somewhat useful for 
benchmarking process costs. U.S. Air Force cost analysts, 
another small group of five, agreed with the cost. analyst from 
the General Services Administration (GSA) that the methods 
were very useful for this purpose. These data described are 
displayed in Table 3. Analyses of variance and regression 
analyses pertinent to question one and discussions of these 
results follow. No other trends or indications were observed 
from the group statistics analyzed. 

Other analyses performed with the data from this question 
were analysis of variance and regression analysis. in 
reviewing the results shown in Table 4, a relationship was 
noted between job classification and the responses for 
benchmarking process costs. A p value of .04 indicates that 


a relationship exists between the responses in the job 
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Classificatiom group: anGdeethelle answers weer benehmanking 
process costs. 


Table 3 
Benchmarking Process Costs Group Data 


Corporate 


Policy 


Program 
miatt 


Industry 
Cost 
Analysts 


USA Cost 
Analysts 
USAF Cost 
Analysts 
GSA Cost 
Analysts 


Source: Developed by researcher. 





A point of interest in this analysis is in the difference 
of opinion between the program managers and other job 
Classification groups. The six program managers who chose to 
respond to this question believed that the methods were vital 
or very useful in benchmarking process costs. The other 
groups felt that the methods were either very useful or only 
somewhat useful for this application. Analysis of variance 
between the question responses and the DoD employer categories 
also showed that a relationship exists. The majority of the 


participants (87), represented by the U.S. Army, DCMC, and 
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DCAA, responded that the methods were somewhat useful where 
the remaining respondents (31) answered that the methods were 


very useful to the process. 


Table 4 
Analysis of Variance For Benchmarking Process Costs 


Between Groups 8 dA, Salk 
Within Groups eo. 30 
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Source: Developed by researcher. 





Regression analysis Lo a felndes question displayed 
relationships between the respondent’s level of mathematics 
education and their answers. The mathematical nature of 
parametric cost estimating methods led to the belief that as 
one’s mathematics education increases so does familiarity with 
the concepts employed in them. This in turn should lead to 
greater confidence in the proper application of the methods by 
these same individuals. There also was a relationship between 
General Service/Senior Executive Service (GS/SES) level or 
military rank for Federal Government employees and their 
responses. In both of these regression analyses, the groups 


observed agreed with the main body of respondents in the 
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degree»ofeutrlitys for™parametrmermeost estimating methods Sn 
benchmarking process costs. These two groups displayed a 
specific trend beyond that initial observation, however. 

The regression analysis for the mathematics education 
relationship yielded a p value of .02 for the six subgroups 
reviewed. This value indicates a relationship does exist 
between the data points, which can be described as weak in the 
following equation: 


fe= — 285 /(X) + 38.3 R* = .033 
(.14) (.40) 


(The values in parentheses represent standard errors of the 
above coefficients.) 


In this analysis, as an individual’s mathematics education 
increases the numerical value of their responses decreases. 
A lower value indicates greater utility because of the inverse 
construction of the response scale used where a one indicated 
the greatest utility and a five the least utility. The small 
value of the coefficient of determination, R* , shows that 
ifieele Of the variation in the dependent variable, responses, 
is due to the independent variable, mathematics education. 

Similar results were observed in the analysis comparing 
the responses with the GS/SES level or military rank. The 
regression equation for this analysis 


Yoeu-445 (4)? Save R? = .134 
ZA, CAEN) 
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Shows that as seniority increases, so does the respondent’s 
opinion of the utility of parametric cost estimating for 
benchmarking process costs. This relationship can also be 
described as weak. 

Although there were a large number of people who did not 
choose to offer an opinion, most of those that did saw some 
promise in using parametric cost estimating methods for 
benchmarking process costs. Groups indicating a high degree 
of utility for them also are typically in an oversight, upper 
Management position or those working to directly support 
others in those positions. These individuals are more likely 
to view a problem in a systems or process approach and may be 
more experienced with using parametric cost estimating methods 
in the area of benchmarking process costs than the group as a 
whole. 

This area of cost analysis is one that may take on more 
prominence in the future with parametric cost estimating 
methods playing a significant role. As DoD reliance on 
military specifications and standards is reduced, the cost 
estimating techniques built on those standards will become 
less reliable. Greater dependence on performance 
specifications may require the DoD acquisition community to 
analyze commercial processes that have had little or no prior 


application in DoD acquisition processes. The DoD Single 
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Process Instlativewe(SPije may sprovaide eoppomtuniticsm tems use 
parametric cost estimating methods to benchmark process costs. 
With supervisory personnel experienced in and amenable to 
their use, parametric cost estimating methods should see 
additional future applications in benchmarking process costs. 

2 Validating Bottoms Up Estimates 

Traditionally, using parametric cost estimating methods 
in this area involves comparisons between the parametric cost 
estimate and the project’s bottoms up estimate. With a long 
history of use in these applications, the aggregate data 
Showed a strong trend toward acceptance of the idea that 
parametric cost estimating methods were somewhat useful to 
validate bottoms up estimates. Only 20 responses were coded 
for no use or a hindrance to validating bottoms up estimates 
while there were 140 responses indicating opinions that the 
methods were at least somewhat useful in these situations. 
Group breakdown data of interest are presented in Table 95 
while Table 6 contains analysis of variance data. 

Examination of the analysis of variance data showed those 
identifying themselves as engineers and program managers 
believed that parametric cost estimating methods were more 
useful than did the other job classification groups. There is 
a Significant difference between the upper level management’s 


opinion of their utility and that of their supporting 
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Organizations. Support groups are represented by the program 
staffer and engineer group data in Table 5. 


Table 5 
Validating Bottoms Up Estimates Group Data 
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Table 6 
Analysis of Variance For Validating Bottoms Up Estimates 


Source: Developed by researcher. 





The program managers and those who set corporate policy 
felt that the methods were vital or very useful in this area 
while the engineers and program staffers thought that the 
methods were only somewhat useful in this functional area. 
This divergence of opinion may be a result of program staffers 
and engineers who had closer involvement with the techniques 


and methods than the program managers and those who set 
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corporate policy. The weak negative relationship exhibited in 
the regression analysis 


Vix) sae Rea wu0S3 
La) (10) 


supports the contention that the upper level supervisors have 
a higher opinion of the method’s utility than do the support 
organizations. The small R* value indicates a weak 
relationship, however. 

Parametric cost estimating methods will continue to be 
used to validate bottoms up estimates. This function is one 
of the traditional uses for parametric cost estimating for 
which there is still a need. A large part of the DoD 
acquisition community is familiar with their application in 
validating estimates derived from other sources. Certain 
Situations lend themselves to the development of bottoms up 
estimates, and the subsequent need to have them validated 
against some standard will always remain. These factors 
should continue to encourage the use of parametric cost 
estimating methods in validating bottoms up estimates. 

S.. Program Estimates 

Developing program estimates is also an area of 
application for parametric cost estimating. Consideration of 
the total costs for a program or system has always been a 
concern and is becoming more important in acquisition 


planning. In the aggregate analysis, the majority of the 
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participants felt that the methods were useful to some degree 
in developing program costs. Only seven respondents answered 
that there was nO use for parametric cost estimating in 
program estimating and no one responded that they were a 
hindrance. The average response value for the entire database 
was 2.25 with a standard deviation of .82. Demographic group 
statistics of interest are shown in Table 7, and Table 8 
contains analysis of variance information. 

In the examination of the analysis of variance data for 
job classifications, program managers, engineers, and contract 
negotiators had somewhat lower group response averages than 
did the remaining job classification groups represented in 
Table 7. Analysis of variance of the DoD component responses 
also showed some interesting trends. The DCAA group’s 
response average was higher than that of every other group 
while the industry group had the lowest response averages. 
The DCAA group opinions are noteworthy as they represent a 
fairly homogenous response from a large subset, 33 responses 
to this question, of all of the responses. DCAA has a 
Significant part in the DoD acquisition process. They have 
published guidance on the auditing procedures for parametric 
cost estimating methods. 

The NASA responses may be an outgrowth of the nature of 


the work NASA performs. NASA has a great deal of experience 


76 


with using parametric cost estimates in numerous developmental 
projects and may be more experienced in their use than the DoD 
organizations. The NASA respondents also have one of the 
highest average levels of mathematics education of the groups 
Surveyed. The DCAA responses may reflect distance from the 
program estimating functions and a different definition of 
what constitutes program estimating than that used in the 
other groups. Also of note is that the DCAA group had one of 
the lowest mathematics education levels of the groups 
Surveyed. These factors may combine to produce the divergence 
@2 Opinion in the utility of parametric cost estimating 
methods between these two groups. 


Table 7 
Program Estimates Group Data 
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Table 8 
Analysis of Variance For Program Estimates 
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The data and statistics for this question appear to have 
contradictory indications. The program manager and program 
staff groups had different opinions of the utility of 
parametric cost estimating methods; the program managers felt 
they were vital or very useful while the program staff felt 
that they were only somewhat useful. The experience and 
mathematics education response values for these two groups may 
explain this divergence. The program managers have a 
Significantly higher mathematics education level on average 
than do the program staff but a lower experience average. 

The relationships between the two regression analyses 
seems to be contradictory. Two of the groups with the most 
experience have widely divergent opinions about the utility of 
the methods in program estimating. One would expect that as 


experience increases so would someone’s familiarity with 
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parametric cost estimating methods, their uses and 


limitations. The experience regression equation 


Y =e 2 6ie@@e - 5.13 R? = .053 
(.08) (Ze) 
indicates a weak positive relationship exists. More 


experience leads to a belief that the methods are less useful 


in program estimating. The mathematics education regression 


analysis 
fee leo) — Jaw Ro = 70383 
CSCa,) (1a) 
shows a weak negative relationship. The DCAA responses 


exemplify these two equations. Their view of the utility of 
the methods in this functional area is low when compared to 
the other groups, and they are one of the most experienced 
groups, but they also have one of lower mathematics education 
averages of the groups. 

The DCAA group had lower opinions of the utility of 
parametric cost estimating methods in this application than 
did the other groups. This may represent a more detailed 
working knowledge of the uses of the methods and their 
limitations than the other groups had. It also may be a 
result of a different view of what is included in the 
definition of a program estimate. These groups may have used 
a more restrictive definition of a program estimate than did 


the remaining groups. In reStreLcuingm@ehe = derinitionmro 
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specific functional estimates produced at various times during 
a program cycle, they may have been more skeptical about using 
parametric cost estimating methods than the program managers 
who may have taken a more expansive view of the definition. 
The majority of the groups in Table 7 show a trend towards 
acceptance of the use of these methods for producing program 
estimates. 

A, Budgeting 

Budgeting iS another area in which parametric cost 
estimates have a long history of use. They have been produced 
for rough order of magnitude budgeting as well as for more 
specific estimates in the budgeting process.. The overall 
database information had all but nine participants indicating 
their belief that parametric cost estimating had some use in 
budgeting decisions. The majority of the respondents chose 
the descriptor very useful as their response to the question. 
The average response value was 2.2 with a standard deviation 
of .85 for the responses. Table 9 displays the group 
breakdown data applicable to this question with Table 10 
presenting analysis of variance data. 

The subgroups identified as cost or price analysts and 
contracting officers were of note in the analysis of variance. 
Both of these groups represent large sections of the database, 


45 and 36 responses to this question respectively, with 
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relatively small standard deviations for both groups. The 
U.S. Army and DCAA subgroups were also identified in the 
analysis of variance as being of note. These two groups also 
represent major parts of the database, and when compared with 
the other DoD component groups, they show relatively tight 
data groupings with response average values larger than those 
of the other DoD component groups. These four groups all 
responded that the use of parametric cost estimating methods 
were of some use in the budgeting process. 

Three other groups, program managers, CONUmACT 
negotiators, and those that set corporate policy, responded 
that the methods are vital or very useful to the budgeting 
process. The responses from the program managers and industry 
are of special interest. Of the groups who would be expected 
to have considerable experience in budgeting, they have the 
highest opinion of the utility of the methods in budgeting. 
The data from the DCAA and program manager groups are 
supported by the mathematics education regression analysis. 
The equation 

¥Y = -.137(X) + 7.33 R* = .051 

(.04) (il 6G. 
indicates that there is a weak negative relationship 
demonstrated in the DCAA high average response to this 


question and relatively low mathematics education average when 
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compared with other groups. The program managers have 
opposite characteristics with one of the highest mathematics 


education levels and low average response to this question. 


Table 9 
Budgeting Group Data 
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Reviewing the results of the analysis leads one to 
believe that parametric cost estimating applications in 
budgeting should continue their long history of use. As the 
data show, there are a large number of groups in the DoD 
acquisition community who feel strongly that parametric cost 
estimating methods are useful techniques for budgeting. The 
increased prominence that they are given through the 


Parametric Cost Estimating Initiative will widen the knowledge 
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about their uses in the community. The increasing accuracy 
and speed at which these type of estimates can be generated 
make them a formidable tool for contingency analyses in an 
environment of funding uncertainty. From their already wide 
acceptance base, this should lead to more applications in 
budgeting in the future. 


Table 10 
Analysis of Variance For Budgeting Data 
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S. Independent Cost Estimate (ICE) Generation 

Generating independent cost estimates iS an important 
part of any acquisition planning process. They may be used in 
any number of program functions to include program estimates 
or budgeting. Parametric cost estimating is one of the tools 
that can be applied to generate these estimates. Responses to 
this segment were centered between very useful and some uSe. 
No responses reflected the opinion that the methods would be 
a hindrance to generating ICE generation. There were a large 


number of no responses to this question, however. The average 
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response value was 2.34 and the median value was 2, or very 
useful. In the analysis of variance the engineers believed 
that the use of parametric cost estimating had the most 
application to ICE generation of the groups. They indicated 
that the methods were very useful while the remaining job 
classification groups felt that the methods were only somewhat 
useful for ICE generation. Tables 11 and 12 contain the 
group specific data and analysis of variance for this segment 
OL Questlonwenc, 


Table 11 
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Table 12 
Analysis of Variance For ICE Data 


Source: Developed by researcher. 






The responses of the engineer job classification group 
indicated the highest level of regard for the utility of 


Paramcertescoct Cormmating Lom proaueing IGES.) 7 This group ie 
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representative of those in the DoD acquisition community 
tasked with ICE generation and who are familiar with the 
process. They also have one of the highest average 
mathematics education levels of any group in the database. 
These factors indicate a viable application for using 
parametric cost estimating methods in ICE generation exists, 
and there is a base of expertise in their use in this area. 

The high regard that the control and USAF groups had for 
the use of parametric cost estimating methods for ICE 
generation is also indicative of their application potential. 
The control group has been studying the use of the methods and 
they have responded that the techniques are very useful in 
this area. Regression analysis revealed the following 
relationship between answers for this question and the 
respondents’ mathematics education levels. 


me —- roo (xj + 8/4 R? = .062 
(0G) (2.14) 


The weak negative relationship between the answers and the 
mathematics education responses supports the observation that 
these groups would strongly support the use of parametric cost 
estimating methods for ICE generation. 

The application of parametric cost estimating methods to 
producing ICEs will continue and become more prevalent. DoD 
Regulation 5000.2-R states, “As defense resources decline, the 


margin available for adjustments narrows, making realistic 
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cost projections all the more critical.” [Ref 44] Parametric 
cost estimating has and will continue to fulfill a major role 
in producing ICES to meet this need. The groups cited in 
Table il are important members of the DoD acquisition 
community who favorably view the application of parametric 
cost estimating methods to ICE generation. With the emphasis 
from the control group on the potential uses of the methods, 
the engineers representing the users of the methods in this 
area, and the auditor group noting that the methods have some 
application in the production of ICEs, they should certainly 
find wider use in future procurement actions and planning. 

6. Software Contracts 

There are a host of databases and computer programs that 
were developed for the purpose of estimating costs and 
schedules for software development. Examples such as COCOMO 
and PRICE-S were cited earlier in the text. The cumulative 
data showed that the largest part of the respondents indicated 
that parametric cost estimating methods had some use in 
software contracts. Few of the participants (27) thought that 
they had no use. The average response value to this question 
was 2.54 and a median value of 3 was observed from the 
aggregate data. 

In reviewing the group data, the USAF was observed to 


have a somewhat higher regard for the application of 
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parametric cost estimating methods to software contracting 
than the main body of the participants. USAF group data, an 
average response value of 2.07 with a .616 standard deviation 
and median value of 2, indicate a view that the methods are 
very useful in this area. The USAF has been greatly involved 
in the use of parametric cost estimates for estimating 
software development costs. The USAF Cost Analysis Agency has 
produced a revised COCOMO model for this express purpose. 
This versatile model can be used to produce a variety of 
outputs throughout the life cycle of a software product. 

Parametric cost estimating has a strong base of 
application as a tool in the area of software cost estimating. 
Estimating the costs of and schedule for software development 
has been a historically difficult area. The increasing 
importance of software in DoD systems and equipment could make 
this a growth area for the future. Parametric cost estimating 
methods should become more useful for software cost estimate 
generation as databases become more interconnected. 

>. Repair Parts Contracts 

Aggregate data analysis presented a view that parametric 
cost estimating methods had some utility in producing cost 
estimates for repair parts contracts. The average response 
value was 2.85, a median value of 3 and a normal distribution 


of the data around that value was observed. Table 13 shows 
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thew eBoOlemecua LOL contracting =officers sands those that set 
Government policy. The groups shown are noted for their 
concurrence within their groups demonstrated by the small 
standard deviations identified for each group.» Both groups 
can also potentially play a major part in the use of 
parametric cost estimating in repair parts contracting. Those 
that set Government policy (through regulation or directive) 
and contracting officers (through applying the methods in 
specific procurement actions) have the potential to directly 
encourage or discourage the use of parametric cost estimating 
methods in this area. That these groups both indicate that 
they see limited use for the methods in repair parts 
contracting would seem to limit their uses here. 

Repair parts contracting is not normally an area in which 
the methods of parametric cost estimating are applied. 
Generally the costs or prices are fairly well established 
through historical prices. The need for estimating repair 
parts cost or prices typically occurs later in the life cycle 
than the stages in which parametric cost estimating methods 
are used. Parametric cost estimating methods could be used in 
ain once Sehedienrs ope repair parts costs s£0r aa system gen 
piece of equipment much more readily than for an individual 


procurement action. The data reflect a preference for other 
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cost or price estimating methods to estimate repair parts 


costs over the use of parametric cost estimating methods. 


Table 13 
Repair Parts Contracting Group Data 
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8. R&D Contracts 

R&D contracting iS an area in which parametric cost 
estimating can be applied in a number of areas. The types of 
items and services procured in R&D contracting are 
developmental in nature, and estimating costs or prices can be 
Cee Cult . The database analysis for the entire response 
group yielded an almost equal distribution between the very 
useful, somewhat useful and no use descriptors, 41, 49, and 40 
responses respectively. An average response value of 2.83 
indicates a somewhat useful viewpoint throughout the DoD 
acquisition community regarding the utility of parametric cost 
estimating methods to R&D contracting. The NASA participants 
had the highest regard for their applicability in the analysis 


of variance. Job classification and DoD component data for 
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the analysis of variance is displayed in Table 15. DCAA 
responses indicated that in their view the methods were of 


some or no use to R&D contracting. 


Table 14 
R&D Contracting Group Data 
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The NASA and USAF data indicate that there are some in 
the DoD acquisition community that feel that parametric cost 
estimating methods can be used in R&D contracting. It is also 
of interest that the technical support group participants were 
the group that had the highest average response rate of all 
the groups. The response rate shown in Table 14 indicates 
that this small group believes that the methods have no use in 
NSD “Comet racer . 

The mathematics education regression analysis supports 
the observations regarding the groups in Table 14. The 
regression equation 


eae COON eer LO R* = .078 
(2 0D) (2g) 
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indicates that there is a weak negative relationship between 
mathematics education and the responses for this question. 
The NASA participants had a higher average mathematics 
education level than did the other groups and they also felt 
that the methods had more use in R&D contracting than did the 
other respondents. 


Table 15 
Analysis of Variance For R&D Contracting Data 
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The application of parametric cost estimating methods 
should continue to be used in this area where the groups 
mentioned have found them useful. Increasing budgetary 
pressures have restricted spending on R&D throughout DoD and 
it is likely that R&D funding will be restricted in the 
future. A lack of useful databases in the R&D field may be 
contributing to the view of almost one third of the 
respondents that parametric cost estimating methods are of no 


use. While parametric cost estimating cannot be used for 
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every instance, the methods do have the potential to produce 
sound estimates for R&D applications. 

oe Service Contracts 

Acquisition of services forms a significant part of the 
Federal Government’s procurement activity. Parametric cost 
estimating methods have not been widely applied to this area 
of acquisition in the past. The majority of the responses to 
this segment were split between some use and no use with 57 
responses for each choice. The average response value from 
the database was 3.09 and the median response value was 3. 
These indicate that the participants felt there was only some 
use for the methods in service contracting. The results of 
the analysis are shown below in Table 16. 

Service ContEaeemng usually does not have the 
characteristics that make it amenable to using parametric cost 
estimating techniques. The procurements in service 
contracting are not often quantifiable in terms of parameters 
that are easily measured and then related to outputs in a 
database. There are few existing databases and software 
products that are used in this area. The control dEams 
average response rate supports this view in their opinion that 
parametric cost estimating methods do not have any use in the 
area of service contracting. The rather large standard 


deviation anateates mam lack Of COnNSensus Within this Gromer 
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however, but at no point does anyone in the group advocate 
that the methods have any more than some use in service 
eoniractiang. 


Table 16 
Service Contracting Group Data 
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fmerers in their opinion but only slightly. Their answers 
lead one to believe that they feel that the methods have some 
use in this area of contracting. Unlike the Baal Group, 
those that set corporate policy have a closer consensus 
regarding the methods’ application in service contracting 
evidenced by the small standard deviation in a group twice the 
size of the control group. 

Defining services to be acquired by performance standards 
may allow greater use of parametric cost estimating methods in 
service contracting. Performance specifications are the 
preferred method of describing supplies or services required 
by the Federal Government. With this emphasis and increased 
awareness of parametric cost estimating methods, the 
techniques may see more use in this area of Federal Government 


procurement. 
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10. Hardware Contracts 

Although hardware can be defined in terms of components, 
assemblies or subassemblies of an electrical, mechanical, or 
electronic piece of equipment, the definition used here is 
more limited. Hardware contracting for this study refers to 
contracting for computer hardware, equipment, or peripherals. 
The rapid pace of development in the computing field leads to 
the use of parametric cost estimating in this area. The data 
and analysis apply equally as well to a broad definition of 
hardware as they do to the narrower definition used in this 
study. 

Database analysis revealed that the majority of the 
respondents answered that the methods were very useful or of 
some use in hardware contracting, 54 and 59 responses 
respectively. Some use was the median response and an average 
response value was 2.55. Contracting officers were the only 
group that showed a consensus, a standard deviation of .62 was 
observed, in their responses to this question. In their 
position, they are also one of the groups that could best 
apply the methods to contracting for hardware. Their 
responses with a mean of 2.65 and median value of 3 indicate 
a view that the methods are of some use in this area. 

While the methods may have application in producing 


estimates to be used in the area of hardware contracting, the 
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drive ctomuse™Commercial—ofif-PMe -Sshel ies lectnology 1 4O.. mane 
Non-developmental Items (NDI) may restrict their uses by the 
Federal Government in this area in the future. COTS items and 
NDI are by definition beyond the stage in which parametric 
cost estimating methods are typically applied in the product 
life cycle, as shown in Figure 1. If the item is truly a COTS 
item or NDI there should be ample pricing data available from 
the commercial environment and less need to apply the 
techniques to this area of Federal Government acquisition. 
This type of acquisition environment may, however, hold some 
promise for the use of parametric cost estimating methods. 
The methods may be used to estimate the costs of integrating 
the various COTS items and NDI that may be purchased to meet 
an agency’s needs rather than for the equipments themselves. 

11. Construction Contracts 

Parametric cost estimating methods have been successfully 
applied to many construction projects. There are plentiful 
data and output parameters in construction that lend 
themselves to using parametric cost estimating methods. The 
CPRG uses a construction example to illustrate the development 
of concepts and CERs in parametric cost estimating. Some use 
was the choice of 54 of the respondents to the survey. There 
were a Significant number of participants (79) who elected to 


not respond to this segment of question one. An average 


2» 


response value for the participants of 2.71 was observed and 
the median response was the some use descriptor. The group 


response data are presented below in Table 17. 


Table 17 
Construction Contracting Group Data 
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Source: Developed by researcher. 










The contract negotiator response is of particular 
interest in Table 17. This group indicated that they believed 
that there was greater utility for parametric cost estimating 
methods in construction contractings than did the cigs 
participants and groups. The contract negotiators indicated 
the methods were vital or very useful in the area of 
construction contracting while others felt that they were only 
of some use in the process. 

The contract negotiators’ consensus on this question is 
also noteworthy when compared with the responses from the 
contracting officers. With similar training and experiences, 
one would expect similar responses from both groups. The 


contracting officers, however, felt that the methods were only 
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of some use in construction contracting. These individuals 
and those that set Government policy can be most influential 
in applying the methods in this area, and both do not see the 
methods contributing significantly in this area. 

Parametric cost estimating methods should continue to be 
employed in estimating for construction contracting. They 
have an established basis for use outside the DoD and a large 
part of the DoD acquisition community involved in construction 
contracting 1S aware of how they can be best applied to this 
area. Base consolidations and relocations could offer 
additional opportunities for using parametric cost estimating 
methods in construction projects. 

12. Commodity Contracts 

Contracting for commodities, such as fuels, clothing, or 
food items, 1S a Significant and vital part of the DoD 
acquisition workload. Commodities may defined as commercial 
items sold in substantial quantities to the general public 
exemplified by the group and class categories in the GSA 
catalogs. Aggregate group analysis showed that the majority 
of the responses to this question were split between some use 
and no use in this area, 51 and 41 responses respectively. 
The analysis also showed a median response value of 3 and 
average response value of 3.23 with a standard deviation of 


./8. The small standard deviation indicates that a consensus 


oF 


on this question exists. No group in the analysis deviated 
fromethiver op. n Lene 

Pricing for commodities is normally accomplished using 
market pricing mechanisms. As a result, there are few 
Opportunities for the use of parametric cost estimates. The 
aggregate group data presented support the idea that the 
methods have limited application in commodity contracting. 
This situation is unlikely to change significantly because 
market prices are one of the preferred pricing methods for 
commodity items. 
Ce SUMMARY 

Question one was broken into 12 segments to take a broad 
view of some of the major areas in the acquisition process to 
which parametric cost estimating may be applied. The results 
of the analysis conducted across these areas showed the 
opinions of the DoD acquisition community regarding the 
applicability of the methods. Overall, the methods were seen 
to be most applicable to the program estimating, budgeting and 
ICE generation and least applicable to commodity contracting. 
In all of the other areas the respondents indicated there was 


some use for the techniques. 
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VI. SPECIFIC APPLICATIONS 


A. PURPOSE 

Questions two through four ask the participants to rank 
the utility of parametric cost estimating methods across a 
series of specific applications in program management and 
contract situations. Question two addresses system life cycle 
concerns as defined in DoD Regulation 5000.2-R. Figure 1 
presents a typical pattern of applying the methods throughout 
the four life cycle phases. Question three presents data on 
the uses of parametric cost estimating methods in specific 
contract type situations. Finally, question four addresses 
applications in pre-award and post-award contract actions. 
B. PROGRAM LIFE CYCLES PHASES 

i. Concept Exploration 

In the first phase of a system’s life cycle, concept 
exploration (CE), a series of short term, independent studies 
are conducted. These studies attempt to evaluate various 
concepts that are proposed to meet a requirement. Parametric 
cost estimating, as shown in Figure 1, is not the primary cost 
estimating method in this phase. The aggregate responses were 
essentially evenly spread across the three utility descriptors 
and no responses. Vital, very useful, some use, and no 


responses garnered 39, 48, 43, and 39 answers respectively. 
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The dispersion of responses indicates a lack of agreement on 
the answer to this question which is also identified in the 
large standard deviation of 1.09. The average response value 
of 2.46, a median value of 2 and a positively skewed 
distribution indicate that the participants felt that the 
methods were very useful in this life cycle phase. 

The job classification analysis of variance data 
highlighted the control and NASA groups. Their responses 
indicated that for this phase they felt the methods had the 
most utility of all of the groups. The auditor job 
Classification group was identified as believing that the 
methods had the least degree of utility in this life cycle 
phase. These results may be an outcome of the experience that 
the control and NASA groups have with this phase of program 
Management and the lesser degree of experience that the 
auditors may have with it. Tables 18 and 19 contains these 
dear 

The low response averages for the program staff, NASA, 
and control groups for this question are noteworthy. These 
groups represent high mathematics education levels and are 
familiar with the actions required in each of the phases of a 
system’s life cycle. The mathematics regression analysis also 
supports this. The regression equation and data 

I OU NOC nao) R* = .132 


(Oe) (eo So) 
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Show a weak negative relationship between the responses and 
the mathematics education level of the participants. These 
groups indicated that they believed that the methods were very 
useful in the concept exploration phase. The data from 
analysis of the auditor and contracting officer Eeanenene show 
that they feel that the methods are only of some use in this 
phase. 


Table 18 
Concept Exploration Group Data 
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Table 19 
Analysis of Variance For Concept Exploration Data 
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The CE phase activity is centered on reviewing various 
solutions to a problem vice analyzing a specific solution. It 
is not until a general course of action is decided on that 
system measures of effectiveness or performance can be 
developed. Without some general definition of how a solution 
to the problem is to be described, a database cannot be 
selected for analysis nor are there any parameters to 
evaluate. Once this has been accomplished, parametric cost 
estimating methods can be successfully used in this area of 
program management. In this phase, they may have more 
application as rough order of magnitude planning tools in 
budgeting or for conducting cost and performance tradeoff 
analyses early in a program than for specific cost or schedule 
estimates. 

2x Program Definition & Risk Reduction 

Activity in the PD&RR phase centers on the evaluation of 
the problem solution identified at the Milestone I decision 
point. Defined in DoD Regulation 5000.2-R, these activities 
may include prototyping, testing, and early operational 
assessment of critical systems, subsystems, and components. 
[Ref 43:p.4] Actions, such as these, are taken to identify 
and reduce risk in the early stages of a program. Responses 
to this segment of question two were concentrated on very 


useful, 60 responses, and some use, 58 responses. An average 
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response value of 2.43 and median answer of very useful was 
observed. 

Analysis of variance results highlighted the corporate 
woLicy, NASA, program staff, and@ control groups “as 
particularly interesting. These groups believe that the 
methods are vital to this phase of a system’s life cycle. In 
fact, the program staff members were unanimous in their 
opinion that the methods were very useful in the PD&RR phase. 
Data analysis results are displayed in Tables 20 and 21. The 
View expressed by the groups that the methods have a great 
deal of utility in this phase is noteworthy. Most of the 
groups work in this area of program management or have 
experience in performing the functions in this phase and 
represent a cross section of upper level management in the DoD 
gfequasation community. 


Table 20 
PD&RR Group Data 
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Source: aaa by researcher. 
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Figure 1 presents the view that parametric cost 
estimating is most useful in this stage of a system’s life 
cycle. This is demonstrated by the response averages and 
standard deviations of the groups shown in Table 20. The 


following regression equations support this observation. 


Mathematics Education 


Y = -—-.201(X) + 9.95 R* = .099 
(aa) a) 

GS/SES Level/Rank 

Y = -.343(X) + 7.09 Bee 94 


ee oy (2.24) 


Although these relationships can be described as weak, they do 
indicate that as one’s mathematics education and rank increase 
the belief in the utility of parametric cost estimating 


methods to PD&RR becomes stronger. 


Taine 7 | 
Analysis of Variance For PD&RR Data 
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Source: Developed by researcher. 


Applications for parametric cost estimating in the PD&RR 
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phase are widespread and well developed. They should continue 
to be used in this phase and become more important as they 
become more accurate and rapidly generated. There is a 
concurrence in the opinions regarding their utility across the 


DoD acquisition community represented by the corporate policy, 
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provide the most benefit. 
oo Engineering & Manufacturing Development 
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(E&MD), one of the functions is to define and. perfect the 
manufacturing processes to be used to produce the equipment 


selected at the Milestone II decision point. These processes 


provide opportunities for the use of parametric cost 
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the defined solution and production of equipment representing 
that solution. 

Participants responded to this question with 62 responses 
in the very useful and 76 responses in the some use descriptor 
categories. There was an average response value of 2.45 and 


the median response was for some use in the E&MD phase. 
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DCAA represented this overaii view that the methods had oniy 
some use in BK&éMD. They were joined in this belief by the U.S. 
Army, a major buying organization in the process. 


In the group data, the responses of the cost estimators 


nna those wh 


and tho 10 set corporate policy were interesting. The 
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estimators believed, as did the DCAA group, that the methods 
have limited application in the E&MD phase while those who set 
sOrporate policy believe that it is Vital to the process to 
use them in the E&MD phase. this divergence of opinion 
between these groups is noteworthy in that all of these groups 


have a major role in program management at this phase. Tables 


22 and 23 contain the group analysis and analvsis of variance 
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E&MD Group Data 
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Dep. Enc & Ope rationas 


The final phase described in DoD Regulation 5000.2-R is 


the Production, Fielding/Deployment & Operational Support 


phase (PF/D&OS). In this phase, production of the equipment, 
system or scitware is begun, and the ocperational use ci uae 


product starts. Application of parametric cost estimates in 
this phase have been restricted. One such use may be in 
verifying that the program is tracking with earlier cost 


estimates by comparing them with actual production costs. 


Responses to this question concentrate on very useful, 52 
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descriptor from the scale used. There were no noteworthy 
group data for this question. The regression anaiysis 
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the experience level of the particinants. The relationship 
between experience and the data iS weak although it daoes 


support the idea that the experienced groups in ihe 
acquisition community view the methods as having limited use 
in the PF/D&OS phase. Few areas of application can 


effectively use parametric cost estimating methods in the 
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of the learning curve concept over the number of units 
produced are the major cost estimating methods’ used. 
Parametric cost estimating methods have little role to play 
and are unlikely to take on greater prominence in this area in 
the future. 

[ne werespons eusatia ys Mam r ieee ele mea: Eerie ome Somerton 
parametric cost estimating shown in Figure 1. The 
participants as a group believe that the methods have some 
applications in the CE phase, most utility in the PD&RR and 
E&MD phases and the least utility in the PF/D&0S phase. The 
groups with the most experience using the technigues are also 
the strongest proponents of their utility in the PD&RR and 
E&MD phases. Future improvements to the techniques that 
increase their accuracy and flexibility should combine with 
the discretion given to decision makers to make sound business 
decisions to increase the uses of parametric cost estimating 
techniques in all of the program life cycle phases. 

C. CONTRACT TYPE 

Ideally, contract type is directly tied to the risk 
inherent in a particular contract situation. These situations 
are in part formed as a function of the program life cycle 
stage that the procurement represents. Figure 1 relates the 
cost risk to the program stages over time. The narrowing cost 


uncertainty band can be interpreted as declining cost risk as 


109 


CnewesoC Lecce samough. themilifemeycle .of a program. The 
contract vehicles typically used in each stage also progress 
from ones that place greater risk on the buyer in earlier 
phases to ones that shift the burden of risk to the producer 
in the later stages of the program. A contract type pattern 


of use throughout a program might be described as [Ref 41]: 


. Phase O = CEs 

e Phase I = erin 

e Phase II - BP bbwOr Crank 
e Phase III - (SAIS (one 5 aie 


One of the goals of program planning is to match.the contract 
vehicle to the risks in each procurement in each phase of the 
program. Figure 8 describes a relationship between contract 
type, risk, and the utility of parametric cost estimating 
methods. 

Question three is divided into segments reflecting each 
contract type depicted in Figure 8. The response data for 
each contract type segment will be examined to determine the 
relationship between this theoretical presentation and the 
reality of the acquisition environment. Group data will be 
presented for the control, program manager, industry, and 
contracting officer gqnuellps Gor each conbwactebype. These 


groups have extensive knowledge about and experience with 
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using various “contract types = "in ditferene™ procimement 


situations. 
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Contract Type Continuum 
Figure 8: Contract Risk Compared With PCE Utility 


Source: Developed by researcher. 
i, Firm Fixed-Price 
Fixed-Price contract arrangements of all types represent 


Ene majority of contracts written. [Ref 39:p.310] In 1994, 
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Eheses@entracts made up 75% of the contracts written by DoD. 
[Ref 45:p.81] Some use was the median descriptor response to 
the question of the utility of parametric cost estimating 
methods to Firm Fixed-Price (FFP) type contracts. This answer 
was chosen by 60 participants in the survey while 55 chose the 
very useful descriptor to describe the utility. Analysis of 
variance of the DoD components supports this opinion of the 
utility of parametric cost estimating in FFP contract 
Situations. Table 24 contains group data for this question 
and Table 25 contains the analysis of variance data. 


Table 24 
FFP Contract Group Data 
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The U.S. Army, DCAA, U.S. Navy, and USAF groups, totaling 
105 respondents, had average response values significantly 
higher than the average. The groups identified in Table 24, 
however, make decisions about which type of contract to use in 


Situations that they are presented with and thus may have more 
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influence in this area. Although there is not a consensus 
across the groups or within any one group, with the exception 
of the contracting officers, the group averages are somewhat 
lower than the average for the entire database (2.56). This 
demonstrates a view that the methods are very useful in FFP 


@enmeracts. 


Table 25 
Analysis of Variance For FFP Contract Data 


DoD Component DF 
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Source: Developed by researcher. 





2. Fixed-Price With Economic Price Adjustment 

Responses to the guestion of the utility of Fixed-Price 
With Economic Price Adjustment (FPE) type contracts was 
primarily that they had some use. The average response value 
was 2.71 and there was a median value of 3 to this question. 
There were also a large number (81) of no responses. The USAF 
respondents had a consensus that they were only of some use in 
these contract situations. Data from the group analysis is 
presented in Table 26. The benchmark group responses reflect 
a higher regard for the use of the methods in situations where 
one would use FPE contracts than did the entire pool of 


participants. 
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Table 26 
Veo ecOnelaece (Group lata 
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oe Fixed-Price Incentive, Successive Target 

The aggregate group response to this question was one of 
some use for the methods in situations that a Fixed-Price 
Incentive, Successive Target (FPIS) contract arrangement was 
Suitable. This question had a large number (83) of no 
responses but of those who did answer, 59 selected some use, 
and 42 selected very useful as their answers. This produced 
an average response value of 2.62 and the median of 3. Aside 
from the benchmark groups, the only group with any type of 
consensus on this question was the Government policy group. 
These participants also felt strongly that the methods were 
only somewhat useful in FPIS situations. The contracting 


officers also echoed Ehisegbelief while the other benchmam: 
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groups felt that the methods were very useful in these 


contract situations. Table 27 presents this information. 


Table 27 
FPIS Contuact Group Data 
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4. Fixed-Price With Prospective Redetermination 

Parametric cost estimating methods have essentially the 
same utility value in Fixed-Price With Prospective 
Redetermination (FPR(P)) contracts as they do in FPIS contract 
Situations. The aggregate response average was 2.60 with the 
median value equal to 3 or some use. Again there were a large 
number of no responses (77) and the distribution was 
positively skewed. Analysis of variance showed that the NASA 
respondents thought that the methods were more useful than did 
the DoD component groups. The data are presented in Tables 28 


and 29. 
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Table 28 
PERM EY e@eneract Group Data 
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Source: Developed by researcher. 


Table 29 
Analysis of Variance For FPR(P) Contract Data 


Source: Developed by researcher. 






Government policy and contracting officer groups differ 
in their opinion of the utility of the methods to these yee 
of contract situations. Both of these groups believe that 
parametric cost estimating methods have limited use in FPR(P) 
contract situations while the remaining groups in Table 28 


believe that they are very useful. 
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Be Fixed-Ceiling-Price With Retroactive 
Redetermination 


The results from the analysis of Fixed-Ceiling-Price With 
Retroactive Redetermination FPR(R) contract type question 
mirror those of the FPR(P) question. The composite group 
response average was 2.64 with the median value equal to 3 or 
some use. There were a large number of no responses (79) and 
the distribution was positively skewed. The control, program 
Manager, and industry groups thought that the methods were 
very useful in these types of contract situations as did the 
NASA participants. These data are presented in Table 30. 

The analysis of variance for DoD oomieermant and job 
Classification, presented in Table 31, highlighted not only 
the NASA group but also the cost estimators, auditors, and 
cost and price analysts. The last three groups listed 
believed, as do the contracting officers, that parametric cost 
estimating methods have only some use in FPR(R) contracting 
arrangements. The opinion of these groups is significant. 
They represent a major part (107) of the participants and a 
cross section of acquisition community who might be expected 


to be involved in applying the methods to a FPR(R) situation. 


le 


Table 30 
HER) mG@ontraee Groups Data 
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Table 31 
Analysis of Variance For FPR(R) Contract Data 
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Between Groups 
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Source: Developed by researcher. 
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6. Fixed-Price-Incentive, Firm 

Group responses to this question were almost evenly 
divided between some use (57) and very useful (52). The 
median response waS some use and the mean response value was 


yO. Tables 32 and 33 show the information for this 


question. 
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FPIF Contract Group Data 
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The control and industry groups felt that the methods had 
more utility for Fixed-Price-Incentive, Firm (FPIF) contracts 
than the other groups that were surveyed. These two groups 
indicated that the methods were very useful while the 
remaining groups felt that they had omy limited 
applicability. The analysis of variance data indicate that 


cost and price analysts, contracting officers, and auditors 
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oO] eee GeSpondents) Ss thoughtmerhbat. ~parametric cost 


estimating methods had only some use. 


Table 33 
Analysis of Variance For FPIF Contract Data 


Source: Developed by researcher. 









7. Cost-Plus-Incentive-Fee 

Cost-Reimbursement type contracts formed a significant 
part (18%) of the contracts awarded by the DoD in 1994. [Ref 
45:p.81] Parametric cost estimating method utility in Cost= 
Plus-Incentive-Fee (CPIF) contract situations was determined 
to be somewhat useful by the survey respondents. An average 
response value of 2.62 and median of 3 was observed. Contract 
negotiators, program managers, industry respondents, and 
engineers (37 participants total), identified in the analysis 
of variance data, thought that the methods would be very 
useful, in CEREeeontuaee Situations. The remainder of the 
respondents (103), represented in Table 34 by the contracting 
officer group, thought that the methods were of only some use 
in this type of situation. It is of interest to note the 
difference of opinion regarding the utility of parametric cost 


estimating methods in CPIF situations between the contracting 
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officers and contract negotiators. One would expect to find 
Similar responses to this question from these two groups 
because of their common training and experience. Analysis of 


Variance data are presented in Table 35. 
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Table 35 
Analysis of Variance For CPIF Contract Data 


source: Developed by researcher. 





8. Cost-Plus-Award-Fee 
The median value of 3 reflects the choice of some use by 


55 participants to describe the utility of parametric cost 
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estimating methods to Cost-Plus-Award-Fee (CPAF) contract 
Situations. The average response value was slightly lower at 
2.59, in large part due to 43 respondents selecting very 
useful as their answer to the question. The analysis of 
variance data showed a clear separation of opinion regarding 
the utility of the methods to CPAF contract situations. 
Contract negotiators, program managers, and engineers (17) 
answered that they were very useful and all of the remaining 
job classification groups (123) had the opinion that they were 
only somewhat useful. 

The analysis of variance information from the DoD 
component groups showed that the USAF had the lowest response 
average and the U.S. Army had the highest average in those 
groups. The data in Table 36 show that the USAF indicates 
that the methods are very useful and the U.S. Army only of 
some use in CPAF contract situations. Analysis of variance 
information is presented in Table 37. The divergence between 
contracting officer and contract negotiator opinions regarding 
this contract type and parametric cost estimate utility 
observed in earlier questions is also noted here. Also 
noteworthy is the consensus that the control and engineer 
groups have on the question. They both believe that the 
methods are very useful or vital in a situation where one 


would use this type of contract. 


Zz 


Table 36 
CPAF Contract Group Data 
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Table 37 
Analysis of Variance For CPAF Contract Data 
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9. Cost-Plus-Fixed-Fee 

There were no groups that displayed a consensus or 
unanimous opinion regarding the use of parametric cost 
estimating methods in situations that would also benefit from 
the use of a Cost-Plus-Fixed-Fee (CPFF) arrangement. Of the 
benchmark groups, only the control and contracting offices 
groups had a consensus of some sort on this question. Table 
38 contains only the question three benchmark groups for 
analysis. 


Table 38 
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The aggregate group average response to this question was 
2.58 and a median response of somewhat useful was recorded. 
The control group had the highest regard for using the methods 
in CPFF contract situations, very useful or vital, and the 
contracting officers the lowest regard for their use in CPFF 
instances. In comparison with the average value for the 


entire database, the control, program manager, and industry 
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groups had sa; muchehiogher regard” Eorvrne utility of paramecurite 
cost estimating techniques in CPFF contracting than did any 
other group. 

10. Time & Materials 

While Time & Materials (T&M) contracts make up only a 
small percentage (6% in FY 94) of the contracts awarded by the 
DoD, a significant dollar value is contracted using this 
vehicle. [Ref 45:p.81] The majority of the responses to this 
question were divided between some use and no use in an area 
that a T&M contract might also be beneficial. The largest 
number of respondents chose some use (53) as their answer and 
49 participants chose 4, or no use to the question. The 
average response value of 3.13 and median response of 3 
indicates that the participants felt that the methods were of 
limited use in these situations. A relatively insignificant 
number (6) of people chose to respond that the methods would 
be a hindrance in situations requiring a T&M contract. 

The benchmark group data and groups identified during the 
analysis of variance are shown in Table 39. Table 40 contains 
analysis of variance results for review. With the exception 
of the program managers, there is an almost universal belief 
that the methods have little application in T&M contracting 
Situations. Program staff opinion differs from the program 


managers in their belief about the utility of the methods 
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here. Program staff respondents are unanimous in their 
Opinion that parametric cost estimating methods are of no use 
in T&M contract situations while program managers indicate 
that they believe that they have some use in the process. The 
groups shown in Table 39 show a belief throughout the DoD 
acquisition community that the methods are of little use in 


T&M situations. 
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Table 40 
Analysis of Variance For T&M Contract Data 


Source: Developed by researcher. 
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11. Labor Hour 

The results of the analysis of the Labor Hour contracts, 
representing 1% of all DoD procurement actions, are similar to 
those of the T&M contracts. [Ref 45:p.81] Again, there was a 
median response of only some use and an average value of 3.13. 
Responses from the benchmark groups in Table 41 reflect a view 
that the methods have little application in this type of 
contract environment. The program staff group is unanimous in 
their opinion that the methods have no use in labor hour 
contract type situations. This differs slightly from the view 
of the program managers who see some application for 


parametric cost estimating methods in labor hour contracts. 


Table 41 
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The analysis of the various contract types indicates that 
there are some contract types that can benefit through 
applying parametric cost estimating methods to the process 
more than others. The average values for the control, program 
Manager, and industry groups indicate a relationship between 
risk, contract type, and the utility of parametric cost 
estimating methods similar to that shown in Figure 8. 
Respondents in these groups viewed the methods as having the 
most applicability in CPFF and least utility with T&M or labor 
hour contract situations. While this relationship exists, it 
may not be as pronounced as that shown in Figure 8. 
Furthermore, the relationship shows a distribution of utility 
that is skewed to the right. Figure 9 graphically displays 


the average values for the various contract types. 


Utility 





FFP FPR(P) CPIF T&M 
FPE FPR(R) CPAF Labor Hour 
FPIS FPIF CPFF 
—— Control ---- Program Managers 
ow Industry ---- Contracting Officers 


Figure 9: Contract Type Compared to Parametric Cost Estimating Utility 


Source: Developed by researcher. 
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The increase observed in the utility lines for the 
control, program manager, and industry groups is not present 
in the contracting officer line. The contracting officers may 
view risk from an individual contract standpoint rather than 
across an entire program. They may see parametric cost 
estimating as a tool to be evaluated for use in one Single 
action and not having any relationship to risks in the 
PoneEract or pregmam. This group is also made up of a 
combination of procuring contracting officers (PCO) as well as 
administrative contracting officers (ACO). These two 
Subgroups may have different viewpoints on the utility of the 
methods based on risk and contract type that are masked when 
they are combined into one group. These differences could be 
a result of the roles both groups have in the acquisition 
process and program management cycles. 

All of the benchmark groups for this question responded 
that there was little or no application for parametric cost 
estimating methods in either T&M or labor hour contract 
Siegations. This type of contract is usually used in 
Situations where the work can be described but not the 
quantity of work needed. Pricing for the labor or materials 
is performed by determining actual labor rates or material 
costs based on the marketplace. These two factors combine to 


make parametric cost estimating methods less efficient in 


129 


these situations than other estimating methods. However, the 
methods should be able to provide some use if adequate 
databases and analysis techniques are available to project the 
required quantities of labor or materials. 
D. PRE-AWARD & POST-AWARD CONTRACT ACTIONS 

Question four asks the participants to offer an opinion 
on the utility of parametric cost estimating methods to either 
pre-award actions or post-award actions. Pre-award activities 
where parametric cost estimating might be useful include 
budgeting, ICE generation, proposal analysis, or preparing for 
negotiations and price analysis. Post-award actions in which 
the methods may be useful could include evaluating an 
Engineering Change Proposal (ECP), analyzing the cost aspects 
of a claim, negotiating a Forward Pricing Rate Agreement, or 
negotiating changes to existing contracts. These actions 
could be performed by either PCO or ACO personnel at various 
points in the acquisition process. 

i Pre-Award Contract Actions 

Pre-award functions such as those previously described 
are the areas in which parametric Rane estimating methods are 
usually enought Sema pe y= mess selrectrly= Given this pre- 
disposition it 1 not surprising to find” that the respons_=e 
show a relatively high degree of utility for the methods in 


pre-award contract actions. The average response value for 
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this questaon iS "2736 q@and™ the median values4is 2 “Gr very 
useful. All of the responses were coded 4 or lower indicating 
that no one felt that the methods were a hindrance in pre- 
award actions. Also, only 13 responses were coded as no use 
for the methods in pre-award applications. Tables 42 and 43 
present the group data and analysis of variance data for this 


question. 


Table 42 
Pre-Award Contract Action Group Data 
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Table 43 


Analysis of Variance For Pre-Award Action Data 
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Aliee Of Sthem ajm@b classification» groups sthamk that 
parametric cost estimating methods are useful in pre-award 
actions but the analysis of variance data highlighted the 
program staff and program manager groups. These two groups 
have a significant difference of opinion regarding the utility 
of the methods. The program managers believe that parametric 
cost estimating methods are vital or very useful in pre-award 
Situations while the program staff are unanimous in their 
belief that the methods are only of some use. The contre# 
group also is unanimous in their opinion. They feel that the 
methods are vital to pre-award actions. Another group of 
interest, the corporate policy group, feels that parametric 
cost estimating methods are very useful in these actions. 
This is also represented in the regression equation that 
relates business size to the responses. This equation 


Yoo eee 6.00 R= 328 
(139 ( ZenOSy) 


indicates that there is a negative relationship between 
business size and the response to this question. As business 
size increases the respondents believe that the methods are 
more applicable in pre-award contract actions. This might be 
expected as the larger businesses have more experience with 
using the methods than the smaller ones do. With this strong 
response, the use of parametric cost estimating methods should 


continue to be used extensively in pre-award contract actions. 
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2 Post-Award Contract Actions 

Parametric cost estimating methods are not usually 
associated with post-award contract actions. Often other cost 
Or pricing analysis techniques based on costs experienced up 
to that point in the contract are used to price changes, 
modifications, claims, or other post-award actions. The 
responses to this question reflect a view that the methods 
have little use in post-award contract actions. The answers 
were almost equally divided between some use (59) and no use 
(53) producing an average response value of 3.04 and median 
response of some use. A relatively insignificant number (8) 
of respondents felt that the methods could actually hinder 
Des@-award contract actions. 

Analysis of variance data showed a relationship between 
the responses and the DoD component and job classification 
groups. These data are presented in Table 45. Group data are 
presented in Table 44. From the analysis of variance, it was 
noted that the control group found the methods to be vital or 
very useful in post-award contract actions as well as in pre- 
award contract actions. Program managers and NASA respondents 
agreed with the control group in believing that the methods 
were very useful in these activities. The DCAA participants 
felt that parametric cost estimating had little if any utility 


in post-award contract actions. 
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The NASA response and their long history of involvement 
with parametric cost estimating offers the potential for 
greater use in this area. This group believes that the 
methods are applicable in certain post-award contract actions 
as do the program managers and the control group. The methods 
should provide the same benefits to a post-award negotiation 
Or acquisition plan review that they do in preparing for 
negotiations in the pre-award phase or in drafting an 


acqu?s Teton yakans 


Table 44 
Post-Award Contract Action Group Data 
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Source: Developed by researcher. 

The DCAA group response has the highest value of any of 
the DoD component or job classification groups. Where the 
program managers are at least one level removed from 
documenting a post-award agreement, the DCAA and contracting 
officer groups are the individuals who bring it to fruition. 


Their belief that the methods are of little use in post-award 


134 


actions is a strong counter to the control and program manager 
groups’ drive to use the methods in post-award contract 


activities. 


Table 45 
Analysis of Variance For Pre-Award Action Data 
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z . SUMMARY 

Questions one through four attempted to bring someone 
participating in the survey through a process of identifying 
that person’s underlying beliefs about where parametric cost 
estimating could be used. The opinions presented in these 
questions address the basic research question of: 

How could a DoD emphasis on using parametric methods in 

cost estimating affect the procurement process? 
The responses of the participants form the foundation of the 
analysis of the remaining questions as they address specific 


subsidiary research questions. 
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The remaining questions, five through ten, ask the 
respondent to address issues regarding the application of the 
methods to specific contract actions. The last five questions 
use a five point scale of agreement or disagreement with each 
statement. A response of one or two indicates agreement and 
those of three or four show ea A response of five 


1s coded as no opinion. 
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VII. ACQUISITION AND SOLICITATION PLANNING 


A. PURPOSE 

The purpose of this chapter is to analyze the responses 
to questions five and six. These two survey questions and the 
first four interview questions address planning areeeS that may 
be impacted if parametric cost estimating methods are used. 
Questions five through ten used a five point scale indicating 
the degree of agreement with a set of statements within each 
question presented. The scale ran between “strongly agree,” 
a one on the scale, to “strongly disagree,” a four on the 
scale, with a five on the scale being “no opinion” regarding 
the question. For questions five through ten, a “no response” 
was coded the same as a “no opinion” response as it was 
entered into the database. 
B. METHODS, MODELS, AND SOFTWARE 

Question five is a six segment series of statements to 
which the respondents indicated that they agreed or disagreed. 
The statements are directed at answering the first subsidiary 
research question, 

What changes could offerors make to their proposal 

preparation, submission and support processes’ to 

accommodate using parametric methods, techniques, or 

software in estimating potential contract costs? 
Responses to each statement will be analyzed in the same 


Manner used in the background questions. A discussion of each 
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Statement’s purpose will be followed by group data analysis 
and presentation of the data. 

abies Acquisition Streamlining 

Question 5a asks respondents to provide their opinion 
regarding the following statement: 

Parametric cost estimating methods, models, and software 

could eventually replace traditional bottoms up cost 

estimating methods in many acquisition estimating 

applications as a result of acquisition streamlining. 
Many acquisition streamlining initiatives are based on the 
principle that business is more efficient in its use of 
resources than is the Federal Government. This statement 
addresses the potential displacement of traditional estimating 
methods by parametric cost estimating methods. The statement 
is directed at determining if acquisition streamlining will 
accelerate this process. The effects of parametric cost 
estimating replacing the traditional methods are addressed in 
later questions. 

The aggregate response to this guestion was one of 
indecision. An average response value of 2.49 was calculated 
from a response distribution of 1(28), 2(62), 3(6/), Mama 
Anon ae This distribution produced a median value of 3 or 
disagreement with the statement. The program staff response, 
shown in Table 46, is very clearly disagreeing with the 


statement. This group does not believe that traditional cost 
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estimating methods will be supplanted by parametric cost 
estimating methods. 

There are other groups shown in Table 46, however, that 
do believe that the methods could eventually replace 
traditional methods in some applications. Engineers and those 
who set corporate policy believe that parametric cost 
estimating methods could be used in many more applications as 
the primary estimating method. Greater discretion given to 
decision makers could result in the methods being applied in 


more situations. 


Table 46 
Question 5a Group Data 
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Figure 10 depicts a situation where a precise but time- 





and-cost-intensive detailed estimate and a somewhat less 
precise but quicker and less costly parametric estimate are 
compared at the same point in the acquisition cycle. [Ref 40] 
Both estimates predict the cost of the equipment will cost 


between $79 and $140. The detailed estimate provides a higher 
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probability thateat willscost ShOS8than any other price inwthe 
estimate range. In some situations, the use of good business 
judgment will require a program manager or contracting officer 
to use the less precise estimate. 

In the past the acquisition community has relied heavily 
on precise, detailed estimates often contributing to program 
delays and added costs. Encouraging acquisition officials to 
make these types of decisions based on sound business 
practices is a critical part of acquisition streamlininge 
Parametric cost estimating is one of the tools available to 
analyze situations and make decisions based on good business 


judgment. 


Detailed Estimate 


Parametric Estimate 


Ad a a Gees 
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Cost 
Figure 10: Estimate Accuracies 


Source: Shields, October 1996 
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Ome Information Systems Improvements 

Through statement 5b, the survey participants are asked 
to consider how technology might affect the cost estimating 
methods that are being used. 

Parametric cost estimating methods, models, anv software 

could replace traditional cost estimating methods as 

larger, more compatible data bases, greater computing 

power, and improved software become available. 
Parametric cost estimating methods can be computer asset 
intensive and improvements in either hardware, software, or 
database management could have dramatic effects on their 
peecision. The participants in the survey agreed with the 
statement that improvements in computer agyoatoe 1a ess could allow 
parametric cost estimating methods to supplant more 
traditional methods. An average response value of 2.38 and 
median response of agreement or 2 were observed for this 
question. The corporate policy group waS unanimous in their 
agreement to this statement. They had an average response of 
1.40 and standard deviation of .548 and everyone in this group 
agreed or strongly agreed. 

Technology advances could potentially change the 
characteristics for the Figure 10 parametric cost estimate to 
more closely match those of the detailed estimate. While the 
performance of parametric cost estimating systems might 


improve with computer technology, advancements there might 
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also benefit traditional methods. The same increasing 
computer capability that could make parametric cost estimating 
more precise could also be used to make bottoms up estimating 
more accurate. [Refs 20,30] Greater connectivity between data 
gathering systems could reduce the cost of -producing a 
detailed estimate just as it may make wetemonn ae cost 
estimating more accurate and faster. The corporate policy 
group response indicates greater acceptance for using 
parametric cost estimating methods in future applications, 
however. 

38 Advantages 

The concepts in the following eratement: eaten the 
motivation for both contractor and Federal Government 
personnel to use parametric cost estimating methods in more 
areas: 

Parametric cost estimating methods, models, and software 

can save contractors and Federal Government oversight 

officials significant time and expense in developing and 

then evaluating proposals. 
The majority (142) of the survey participants agreed with this 
statement. The median response was 2 and the average response 
value observed was 2.04. The group data presented in Table 47 
show a range of activities across the acquisition community 
that believe that parametric cost estimating methods have the 


potential to speed up the acquisition process and reduce the 
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costs of proposal preparation and evaluation. The control, 
engineer, and corporate policy groups strongly feel that these 
methods can provide the advantages listed. This indicates 
that there is a willingness to use the methods on a wider 
basis. The belief and acceptance by DCAA of the potential 
that the methods could lead to greater Senos FOr Mas img 
the methods in more areas where traditional cost estimating 


methods have been used. 


Table 47 
Question 5c Group Data 
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4. Data Collection 

Question 5d asks the participants to record their 
Opinions about the following statement: 

Parametric cost estimating methods, models, and software 


could significantly change the way potential contractors 
collect and summarize cost information. 
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Many cost collection systems primarily used by Federal 
Government contractors are designed or have been modified to 
colbatke information MEO. suppemEieinvoicingywes in additions 
providing management information, one of their secondary 
functions is to provide data for their cost estimating 
systems. Data available from these systems form the databases 
from which parametric cost estimates may be produced. This 
statement addresses the concern that these systems may not be 
ideally suited for producing information to be used in 
parametric cost estimating systems. 

The participants agreed with the statement that 
parametric cost estimating could affect the way cost data are 
collected and presented. A median value of 2 and an average 
response of 2.14 were noted. An analysis of variance 
performed on the DoD component response data showed that the 
NASA and USAF groups felt more strongly in agreement with this 
question than any of the other groups. The remaining DoD 
component groups were clustered around simple agreement with 
the statement. Within the USAF group, the USAF cost analysts 
were notable for their strong belief that the methods could 
change the way cost data were collected and summarized. Group 
response and analysis of variance data are presented in Tables 


48 and 49. 
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Table 48 
Question 5d Group Data 
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Table 49 
Analysis of Variance For Question 5d Data 


Source: Developed by researcher. 














Se Cost Overruns 

Most of the survey participants disagreed with the 
following statement: 

Parametric cost estimating methods, models, and software 

Should decrease the probability of cost overruns where 

used. 
A total of 110 respondents disagreed with the statement which 
produced a mean response value of 2.89 and disagree (3) 
response for the median. The analysis of variance on job 


erassiftrvcation responses showed almost overwhelming 
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disagreement with the statement. The auditor group data in 
Table 50 represents that view. 

Two groups in the analysis of variance were noteworthy 
because of their agreement with the statement. Table 51 
contains the analysis of variance data. The program manager 
and engineer groups feel that parametric cost estimating 
methods may have some use in preventing cost overruns. 


Table 50 
Question 5e Group Data 
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Table 51 
Analysis of Variance For Question 5e Data 


Source: Developed by researcher. 






There is also a relationship between mathematics 
education and the responses that supports the group data in 


Table 50. The regression equation 
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YY = = "eee x) +6740 Roe= 907 
(.04) (l6.G,) 


indicates a weak negative relationship between the responses 
and mathematics education. The greater level of mathematics 
education that both groups have and the experience that the 
program managers have may have produced this result. These 
two groups may view the issue of cost in terms of a range of 
possible costs, similar to that presented in Figure 10, that 
parametric cost estimating methods may help to define. Given 
the flexibility to work within that range may provide a way to 
manage risks to avoid cost overrun situations. 

6. Cost and Pricing Data 

Given the response to statement 5d, one would also expect 
a Similar response to the following statement: 

Parametric cost estimating methods, models, and software 

should change the way cost and pricing data submitted to 

Support a proposal is viewed. 
Cost and pricing data are “... all the facts that can be 
reasonably expected to contribute to the soundness of 
eoemmeces Of future costs.” [Ref 13:part 15.801] If the 
methods used to collect and summarize data are changed by a 
potential contractor then it is reasonable to assume that the 
cost and pricing data that the company submits to the 


Government in a proposal will also change. 
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The average response to this statement was agreement. An 
aggregate group average of 2.29 and a median value of 2 was 
noted. A broad cross section of the DoD acquisition community 
is presented in Table 52 who agree with this statement. 
Individuals who prepare estimates for both the Federal 
Government and the DoD contractor community are represented in 


agreement with the statement. 


Table 52 
Question 5f Group Data 
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Source: Developed by researcher. 


to receive and then analyze a proposal to a solicitation which 
could include cost and pricing data. All of the groups listed 
show a consensus viewpoint. The contract negotiator group is 
also unanimous in their agreement with the statement. 

The responses to this series of statements indicate that 


the commercial industry component of the DoD acquisition 


148 


community 1s willing to US@maparametrig. CoOSe® yest imabing 
methods. The data also indicate that many in the Federal 
Government are willing to accept the use of the methods also. 
To take this step, both parties should agree that the 
contractor community may have to take several actions. 
Generating cost estimates based on parametric methods will 
require investments in software either for database analysis 
and the generation of models or for the purchase of existing 
models that can be modified. The benefits that might drive 
this change would accrue to both Government and industry in 
the reduced acquisition cycle times, lower cost proposal 
preparation costs, and greater management flexibility. digs 
will also require some changes to the systems used to collect, 
collate, and analyze cost data within industry and the 
Government. For a contractor organization that elects to use 
parametric cost estimating methods as a primary tool for 
generating data for contract management, significant changes 
to his existing structure could be required. 
Cx SOLICITATION PLANNING 

Question 6 addresses the issues raised in the second and 
third subsidiary research questions in seven parts. The 
questions it deals with are, 


1. What impacts could emphasis on using cost estimates 
based on parametric cost estimating techniques have on 
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source selection and on the evaluation of proposals by 
the DoD? and 


2. How could source selection criteria and evaluation 

factors be constructed to fairly evaluate a proposal that 

uses a parametric estimate with one that does not? 
Providing a level playing field for all solicitation proposals 
1S a primary goal of Federal Government contracting. The 
statements in this question deal with how the Government can 
construct that level playing field within the solicitation it 
issues in regard to the use of parametric cost estimating 
methods. 

iL. Encouraging Parametric Cost Estimating Use 

The first statement in this question asks the respondent 
if the Government should simply encourage the use of 
parametric cost estimating methods. 

In solicitations, buying activities should encourage the 

use of parametric cost estimating for specified 

procurement actions. 
The majority of respondents overwhelmingly agreed with this 
statement, 95 individuals selected agree, and 27 selected 
strongly agree. An average response value of 2.21 and a 
median value of 2 was shown in the cumulative data. The group 
data in Table 53 is a display of groups within the DoD 
acquisition community whose responses are noteworthy for this 


question. 
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This diverse “qroup all aqrea that “the Veroposimng 
organization should be encouraged to use the methods where 
applicable. As one might expect, the group representing 
industry is most emphatic in their agreement with this 
statement. Given the present environment of preference for 
disengagement from intrusive inspection systems and burdensome 
requirements, this is the most likely form of emphasis that 


DoD will use to promote parametric cost estimating techniques. 


Table 53 
Question 6a Group Data 
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28 Specifying Methods, Models, or Software 

Question 6b, shown below, deals with more aggressive use 
of parametric cost estimating methods by requiring them to be 
used when the buying organization feels that they would be 
helpful. 

In solicitations, buying organizations should, specify 

parametric cost estimating methods, models, or software 

LSOGRMESS ER OT OPCS a ConsSeructlon: 
The average response to this statement was to disagree. An 
average summary value was 2.73 and the median value of 3 
indicate this sentiment. The three groups whose data are 
shown in Table 54 all disagree with this statement. All three 
groups’ responses are noteworthy; the control group because of 
their long study of the appropriate uses of parametric cost 
scene methods and both contracting officer groups because 
of their experience in generating solicitations. 

Regression analyses of the GS/SES level and business size 
responses show a relationship exists. The equations and 
Statistics 


GS/SES Level 


Y = .286(X) - 1.45 Re) lO 
(214) ine 90) 

Business Size 

Y oS (ef ( x) = 7 Res 
(.20) (Zhe) 


demonstrate positive relationships between responses and 


increasing Government rank and also increasing business size. 


152 


The more senior the respondent the more they disagreed with 
the statement and the larger the business the more that they 
disagreed with the statement. It is very clear that neither 
the Government nor industry wants the Government to specify 
cost estimating methods to be used in response to a 
solicitation. 


Table 54 
Question 6b Group Data 
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Question 6c records participant opinions about the 
statement that follows: 
In solicitations, buying activities should leave 
parametric cost estimating methods, models, or software 
Seleceton tO, CONEraGEOr d1semetion. 
Leaving the use of cost estimating methods to be used 
completely up to the seller’s discretion was agreed to by 111 


of the respondents. This produced a median value of 2 and an 


average response value of 2.33. The analysis of variance with 
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the DoD component data showed a relationship was present in 
Ene Senreismat Tom. VOL the sDoD job) classification gueupssenly 
the NASA and USAF groups, 22 participants, clearly agreed with 
this statement. The data to support this are shown in Tables 
Samand 564 

With the exception of the NASA and USAF groups, all of 
the other job classification groups were split between 
agreement and disagreement. The DCMC response data in Table 
95 is representative of these groups. Unexpectedly, there was 
also no consensus among the industry participants regarding 
this statement as there was in statement 6a. 


Table 55 
Question 6c Group Data 
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Table 56 


Analysis of Variance For Question 6c Data 
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Source: Developed by researcher. 
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4. Required Cost Breakdown Formats 


Respondents to question 6d were asked to provide their 
opinions about how parametrics might change data presentation 
for solicitations through the following statement: 

bo SsOlICIEatlons, buying “agtixeries should require a 

specific cost breakdown format for parametric cost 

estimate based proposals. 
This statement addresses the issues in the question: 

If parametric cost based estimates are used in a 

proposal, should the buyer state the format that the cost 

information is to be presented? 
With so many different systems producing estimates, some 
standardization may be helpful in evaluating smareoeaule ina 
timely manner. A specified cost breakdown format is one 
method of standardization that 1s available. 

The majority (108) of the respondents agreed to this 
premise producing an average response value of 2.35 and median 
response of 2. Analysis of variance on the job classification 
responses showed that a relationship between the responses and 
the classifications was present. Tables 57 and 58 contain the 
data pertinent to the question 6d statement. The auditor, 
program staff, program manager, and contract negotiator 
groups, 54 respondents total, all indicated agreement with 
this statement. These groups might all be part of a source 


selection team and see a value in having a specified format. 
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Another significant group in the source selection process, the 
COnbrderemng Officers, did not come’ to a consensus on either 
agreement or disagreement with the statement, however. 


Table 5/7 
Question 6d Group Data 
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Table 58 
Analysis of Variance For Question 6d Data 
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Regression analysis with the GS/SES level and business 
size group data showed positive relationships in the data. 
The equations 

GS/SES Level 


Yok ce OOM) a Ro =. iw 
(.14) 2-019) 


ho 
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Business Size 


Y = .484(X) - 4.61 Rivmeaw2 
(ee lice) (2.45) 
describe these relationships. Again, as one’s rank or 


business size increases so does the response value to the 
question. These results indicate that experience with the 
methods would lead to allowing the seller discretion in not 
only their use but in how they choose to present the data from 
those estimates. 

Be Cost and Technical Proposal Relationships 

A second area of concern in evaluating proposals is that 
of having the information in each section correspond to one 
another. The following statement addresses this issue: 

In solicitations, buying activities should require a 

specified format to extract labor hours, subcontractor, 

or material quantity estimates for technical proposal 

evaluation. 
A standard format required by the buyer stating that format 
Sema provide that concurrency. Similar results to the 6d 
statement were observed. An average response value of 2.37 
and a median value indicate agreement with the statement by 
the entire group of participants. Diem orOmp “COE econurace 
negotiators had a consensus that they strongly agreed with 
this statement. Their group response was the only group 
response to reach any type of consensus of all of the 


demographic categories identified. 
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6. Statistical Evaluation Criteria 

Determining what criteria will be used to evaluate the 
validity of the CERs used to produce estimates included in a 
proposal and stating them in the solicitation is another 
method of creating a level playing field. Statement 6f, as 
shown below, examines this area: 

In solicitations, buying activities should state criteria 

used to evaluate the statistical significance of cost 

estimating relationships used to support a parametric 

cost estimating model. 
A great deal of agreement was noted for this statement. The 
average response value was 2.06 and the median value was a 2, 
largely a result of 137 respondents electing to.either agree 
or strongly agree with the statement. The group data in Table 


59 show a wide range of acquisition activities across the 


entire acquisition community who agree with the statement. 


Table 59 
Question 6f Group Data 
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Information regarding CER validity could be interpreted to be 
required by FAR part 15.406-5 in that it could be defined as 
a Significant factor that might be considered in awarding a 
Contract. 

bls Traditional or Parametrics Methods in Solicitations 

Statement 6g, show below, asks survey participants their 
opinion about the concept of fairness in a solicitation if 
parametric methods are either specified or excluded. 

In solicitations, buying activities should require cost 

proposals based on either parametric cost estimating 

methods or traditional methods. 
A source selection plan should not favor one proposal over 
another simply by the way the proposal evaluation plan is 
constructed. The plan must make sure procedures do not act 
to exclude or discourage potential offerors from submitting 
proposals. The statement addresses the problem of fairly 
evaluating a parametric-based cost proposal with one produced 
uSing more traditional detailed methods. Can an evaluation 
plan be constructed that will fairly evaluate both types of 
cost estimates or must the buyer specify which an offeror 
Should use because they cannot be evaluated fairly? 

There was no agreement on this question at the level of 
the entire pool of participants nor on an individual group 
level. The aggregate response average value was 2.55 and a 


median value of 3 was observed. These data indicate that the 
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group disagreed in general with the statement but that they 
were effectively split over agreeing and disagreeing. There 
were also a Significant number (51) of no opinion or no 
responses to this question. 
Dez SUMMARY 

The results from the analysis of the statements in 
questions five and six provide a number of areas that could 
be affected by the use of parametric cost estimating methods. 
Chief among these areas is the contractor cost estimating 
system. The systems will respond to the encouragement that 
the Federal Government members of the DoD acquisition 
community provides. There was universal agreement that buying 
activities should encourage the use of parametric cost 
estimating methods but not require them. 

One of the most effective methods of encouraging the 
use of the methods is in the source selection process. An 
evaluation of a parametric-based proposal that is viewed by 
the offeror to be unfair could be taken as discouragement. 
The trend should remain with telling the offerors what is 
required and letting them propose methods of achieving that 
goal. In that regard, source selection plans should be 
constructed to allow any reasonable type of cost proposal 


submitted to be fairly evaluated. 
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Required formats for data submission in terms of the 
cost proposal or technical proposal were viewed by the group 
as useful in creating a level playing field. These actions 
have the advantage of letting the offerors meet the goal that 
1s required rather than telling them how it is to be 
accomplished. They prevent the Government from dictating 
specific methods to be used while every offeror has some idea 
of how the information provided will be used. They have the 
added benefit of providing some consistency throughout the 
proposals which speeds up the evaluation process. 

If parametric estimating methods are acceptable for 
use as a basis for a cost proposal, the proposal must state 
how they will be evaluated when compared with other methods. 
FAR part 15.605(c) (d) (1) addresses the issue of evaluation 
factors under which this issue falls. 

»-.- at a minimum, the solicitation shall clearly 
state the significant evaluation factors, such as 
cost or price, cost or price-related factors, past 
performance and other non-cost or non-price-related 
factors, and any significant subfactors, that will 
be considered in making the source selection. [Ref 
mespart 15.605] 
EeeouyiIng Organization must consider constructing their 
solicitation to include the possibility of parametric 


estimate-based cost proposals if all proposals are to be 


evaluated fairly. The members of the source selection 
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evaluation board should also consider this issue as cost is 


always an evaluation factor. 
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VIII. PROPOSAL ANALYSIS AND NEGOTIATIONS 


A. PURPOSE 

Chapter VIII is an analysis of the responses to questions 
seven through ten. These survey questions and the last four 
interview questions address issues that may arise during the 
evaluation of a parametric-based proposal and subsequently the 
negotiations that may result. The four survey question 
responses were graded along the same five point scale used in 
questions five and six. With the exception of question nine, 
all of the questions were a series of statements upon which 
participants recorded their opinions. 
B. COST PROPOSAL SUPPORTING DATA 

Question seven is a three part series of statements that 
again address the first subsidiary research question. 

What changes could offerors make to their proposal 
preparation, submission, and support processes’ to 
accommodate using parametric methods, techniques or 
software in estimating potential contract costs? 

In the three statements for question seven, an offeror has 
decided to provide a parametric-based cost estimate in 
response to a solicitation. The question addresses the 
information that might be submitted to substantiate that 
estimate. The Government goal in obtaining information about 
the estimate centers around regenerating that estimate to 


understand how it was generated, its limitations, and 
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strengths. Beoveding: data to domthis sera vital part of 
submitting a successful proposal. 

Ale Software and Input Data 

Statement 7a addresses one of the ways that an offeror 
could chose or be required to provide supporting information 
for a cost proposal. 

A potential contractor should provide the parametric cost 

estimating model software and input data used to support 

a cost proposal. 
Providing the software and input data used to produce an 
estimate is one way to meet the Government goal of being able 
to reproduce the estimate. Without these tools it cannot be 
accomplished. There was a strong agreement to this statement 
in the responses. The response average was 1.81, generated in 
large part by the large number of people who answered strongly 
agree (66) and agree (95) to this statement. Responses to 
this statement also had a small standard deviation (.76) 
indicating a consensus among the respondents on this 
statement. The group data in Table 60 provides a sample of 
the groups that agreed to this statement within the DoD 
eacGuasletLon COMMUNE Y . 

Agreement to this statement covered the entire 

spectrum of functional areas that might be tasked to evaluate 
a cost proposal. All responded that this information should 


besprovided by an Offeror in SWppert Of a COSt proposal ia- 
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industry response to the statement was not as clear. There 
was no consensus in the industry response as a whole nor in 
the group that set corporate policy. 


Table 60 
Question 7a Group Data 
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a. Justifying Parametric Use 

Beyond providing the software and input data, there 
should be a supportable reason to use parametric cost 
estimating methods. They should be applicable to the 
procurement action in question and the work relatable to the 
databases used. Shown below, statement 7b examines this 
issue. 

mB potential contractor should previdesgust#eaeation to 

Support the use of parametric cost estimating models and 

Mmputs Gata I Wise GOSt DEOpOSal. 
The majority of the participants answered that they agreed 


(107) or strongly agreed (64) with the statement. The average 


response value was 1.75, a standard deviation of .65 was noted 
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and agree was the median response. The widespread agreement 
to this statement includes almost every functional group in 
the DoD acquisition community. The major functional areas of 
the community that typically are involved in evaluating 
proposals are shown from buying organizations to the industms 


All of the groups shown in Table 61 have a consensus on the 


issue. 


Table 61 
Question 7b Group Data 
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Ss Data and Rationale 

Additional information that is critical to understanding 
a parametric cost based estimate is knowing how it has been 
validated for use. Statement 7c is presented below followed 


by the analysis of the data collected regarding this issue. 


166 


A potential contractor should provide data and rationale 
used to validate a parametric cost estimating model for 
a specific set of circumstances in support of a cost 
proposal. 
The statement indicates the offeror should provide information 
about how the model was modified, the database updated, and 
how often these tasks were performed to give the buyers the 
level of understanding that they need. The groups shown in 
Table 62 again show the level of agreement with this statement 
across the DoD acquisition infrastructure. 


Table 62 
Question /c Group Data 
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Agreement to this statement was also strong and widely 
represented in the DoD community. Strongly agree was selected 
by 71 and agree by 106 respondents leading to an average 
response value of 1.67, a small standard deviation of .61 and 
a median value of 2. 

Offerors should provide information described in the 
three statements in question seven. Gathering these data 
could require new information systems to be used in industry 
to capture, collate, and display this information. The 
responses show support from industry to provide this 
information as well as a desire by those charged with 
evaluating proposals to have these data available. 

C. DCAA SUPPORT 

After a proposal is submitted, it is then analyzed in 
accordance with the evaluation plan. In a four part series of 
statements, question eight addresses possible functions that 
may need to be performed and who is best suited to do them. 
These statements address issues raised in the second 
Subsidiary research question. 

What impacts could emphasis on using cost estimates based 

on parametric estimating techniques have on _ source 

selection criteria and on the evaluation of proposals by 
the DoD? 
Each statement assumes that DCAA is the best organization to 


perform the task identified. The respondents are asked to 
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fill in another organization if they feel that DCAA is not the 
appropriate action agency for the function in question. 
Information about the written responses will be provided with 
the analysis for each question. 

ib, Establishing Validity Metrics 

Determining the validity of a parametric model to a given 
set of circumstances is critical to understanding the outputs 
of that model, their uses, and limitations. Statement 8a, 
Shown below, addresses this concern. 


DCAA should establish metrics to determine a parametric 
cost estimating model’s validity in a given situation. 


Defining measurement criteria to gauge the effectiveness of 
a model in a particular situation refers to the CONCeEPEnOu 
calibration defined in the Parametric Cost Estimating 
Handbook. If models are modified for use, descriptions of how 
they were changed to reflect the situation in which they will 
be used should be provided to proposal evaluators. This 
statement asks the respondent to determine who should set 
forth these criteria. 

There was a large amount of agreement with this statement 
indicating that DCAA should perform this function. The 
average response value of 2.25 results from the large number 
of participants (121) agreeing with the statement. The 
program staff, U.S. Army cost analyst, and USAF contracting 


officer groups represent agreement within the survey 
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participants. This is contrasted with the disagreement that 
the control group expressed. The statement group data are 


shown in Table 63. 


Table 63 
Question 8a Group Data 
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Regression analysis also showed a disagreement in the 
responses. The business size regression equation 
Business Size 
Yoo. (Sox) we 7 26a Rois 5d: 
(a) (28230) 
indicates that as the business size increases so does the 
disagreement with the statement. The same analysis performed 
with the GS/SES level data shows an opposite relationship. 
The equation and data 
GS/SES Level 
Y = -.418(X) + 8.28 R? = .23 
(olay (1.84) 


show a negative relationship. As the Government rank rises so 


does the level of agreement with the statement. The division 
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is clearly along a Government-Industry split in the 
Bequisition community. 

There are two areas that may be the source of the 
disagreement. The respondents may feel that the function does 
not need to be performed or that DCAA is not the appropriate 
Organization to do the work. Since the functions described 
appear as a central concept in the Parametric Cost Estimating 
Handbook and several papers written by members of the control 
group, the likely source of the disagreement is in the area of 
who should do the work. Two answers were written in by the 


control group. Three people recommended that either 


is DEMC sheuld performethis function;e4(2) ciethat 
2 Tiis wormeeshould be@leSt to@the contwactor’s 
discretion. sip 
These two responses were also prominent in the aggregate of 
written responses to this statement. 
In all of the surveys returned, six groups other than 


DCAA were identified to perform this function: 


1. Buying Organizations (14) 
2. Offerors (12) 
Be DENG 15) 


4. Cost Analysis Improvement Group (CAIG) (2) 


a 


oe A DCMC, DCAA, BuyangeOrganization, and Offeror Team 
(2) 


6. Program Managers (1) 


The majority of the added recommendations concentrated on 
DCMC, the buying organization, or the contractor performing 
the work. DCAA and the control group recommend leaving this 
to the contractor’s discretion rather than assigning it as a 
DCAA responsibility. DCMC and the buying organizations, USAF, 
U.S. Army, and U.S. Navy, would prefer to take on this 
function rather than task DCAA with it. 

Both possible issues of contention in this statement are 
addressed by Dennedy, Deutsch, and Hertling in their article. 
They state that this function must be accomplished and 
recommend that it be done prior to using the models to 
generate estimates. [Ref 8:p.1] They also directly address 
the topic of who should perform this verification activity for 
models generated by the potential offeror. 

The parametric model (or CER model) should be 
developed at the local level in coordination with 

the cognizant ACO and DCAA auditors. Ideally, a 

team made up of ACO pricing, buying office 

representatives and the contractor’s parametric 
model builders should develop the model jointly in 

an integrated product team (IPT) process. The 

joint team should review and validate the database 

upon which the model is based, the logic and 

statistical validity of the model itself and how 


the model will fit into the contractor’s estimating 
system. DCAA, aS an independent examiner, will 


Li 


provide .input,....on the ggeentmaeror’ seicenmnals- 

database, and updating procedures. [Ref 8: p.1] 
In this way, the teaming concept and IPTsS are introduced at a 
very early point in the procurement process. The write-in 
responses indicating that a team would be beneficial came from 
both DCAA and DCMC for this segment of question eight. While 
teaming may be the best solution to the problem of who should 
validate and how a model should be validated, the buying 
organizations do not seem to have considered that idea at this 
point in responding to this statement. 

Ze Updating Models 

A second facet of model validation concerns when the 
model is modified or database updated and who should make this 
decision. Survey participants were asked to record their 
Opinions to statement 8b: 


DCAA should establish criteria to determine when a 
parametric cost estimating model should be updated. 


The same basic issues that are evident in the concept of 
calibration apply to when a model should be calibrated. The 
participants agreed to this statement in the majority; 126 
people either strongly agreed or agreed with the statement. 
An average response value of 2.20 and median response of 2 was 
observed. The data shown in Table 64 represent the agreement 
throughout the acquisition community on this issue. The 


unanimous agreement in the program staff group and degree of 
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Consensus re@ehead on this issue in the other groups 4s 
noteworthy. Industry and Government members of the 
acquisition community agree that someone should set criteria 
to determine when models should be updated. A significant 
portion (20%) of the respondents recommended that someone 


other than DCAA be tasked with this function. 


Table 64 
Question 8b Group Data 


aoe 
Staff 


Coniernol 
Group 


OS s enuemny 
Cosie 
Analysts 


Industry 
Cost 
Analysts 


Contracrmng 
Officers 


Source: Developed by researcher. 





With the exception of the program managers, the list of 
other recommendations is the same as in the previous 
statement. The buyer (9), offeror (5), DCMC (18), and a team 
arrangement (4) were the groups most often recommended to 
perform this work. Of the responses from buying 


organizations, the write-in responses recommended either that 


174 


they should determine when a proposed model requires updating 
@—@ that DCMC . take On maeaat work. The DCAA written 
recommendations indicate that the offeror should perform this 
work. 

The Dennedy et al article does not specifically address 
the issue of when a model should be updated but the concept is 
included in the duties listed for the team members. Inherent 
in the article is an assumption that the offeror has taken the 
initiative to keep the model current and that it is applicable 
to the given situation. The Government functions in a review 
and evaluation mode with three organizations, DCAA, DCMC, and 
the buying office, performing various review duties. The 
currency of the model is considered in the: 

1. DCAA review of calibration and validation of the 
EoOlS, 


2. DCMC technical review of the contractor’s cost 
modeling technique, and 


3. Buying office review of the offeror’s assumptions 
made in the model. [Ref 8:p.3] 
Ideally, this would be performed in the team environment 
presented in the previous statement analysis. However, the 
written responses to this question indicate that the buying 
organizations, DCMC and DCAA, do not view teaming as an answer 


to this problem. 
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SA Model Validation 

The issues addressed in statement 8c, shown below, are 
centered around the concept of model validation. 

DCAA should verify that a parametric cost estimating 

model accurately predicts costs for the expected contract 

work. 
While there is no way to be completely certain, until after 
the work is done, that a model has accurately predicted the 
costs for a project, there are methods that can be used to 
determine its accuracy. This process is described in detail 
as validation in the Parametric Cost Estimating Handbook. It 


may include: 


1. Calibration of the model to historical cost data, 
2. Estimating the cost of past completed projects, and 
3. Comparing these estimates with actual costs to 
demonstrate acceptable accuracies, [Ref 37:p.18] 
as steps in a validation process. The average response value 
of 1.93 and median value of 2 show a strong level of agreement 
with the survey statement. The strong support for the 
statement is shown throughout the survey respondents in the 
group data presented in Table 65. Program personnel, 
contracting officers, and industry groups all agreed that DCAA 
should perform this verification. That a smaller number of 


written recommendations was received for this statement 
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indicates agreement that DCAA is also the organization best 
suited for doing this work. 


Table 65 
Question 8c Group Data 
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Dennedy, Deutsch, and Hertling also state clearly 
the function should be performed by DCAA but they emphasize 
that ideally it would be done in a team environment. [Ref 
8:p.3] The results of the recommendations received show that, 
again, buyers would prefer to retain this work or task DCMC 
with it. DCMC written recommendations are that they would 
prefer to retain it. It should be noted that only 15% of the 


respondents chose to provide a written response. Of the total 
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number of written recommendations to this statement, DCMC 
provided six. DCAA is well-suited to perform this task in 
their traditional role, and the lack of DCAA written responses 
shows an agreement to take on this work. 

4. Model Operation 

The basis of analyzing a cost proposal is understanding 
the proposal in detail and the relationships within it. Using 
an offeror’s model to generate cost estimates is one method of 
applying the information provided about that model to ensure 
this level of understanding. The final statement of question 
eight asks the respondents to give their opinion about the 
issues in this statement: 


DCAA should run a contractor’s parametric cost estimating 
model for purposes of analyzing the cost proposal. 


There was again a great deal of agreement with DCAA being 
tasked with operating an offeror’s model for these purposes. 
There were 129 responses in agreement with this statement 
which produced an average response value of 2.08 with a .74 
Standard deviation and a median value of 2. Table 66 presents 
the group data for this statement. Notably, the contre 
program manager, and program staff groups were unanimous in 
their agreement with the idea that DCAA should provide this 
service. One major buying organization, contracting officers 
and three job description groups reached a consensus that this 


service should be provided. 
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Table 66 
Question 8d Group Data 
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Written responses to this statement show that buyers and 
DCMC would prefer to perform this work themselves. The lack 
of DCAA written responses would indicate that they agree with 
their organization being tasked with this function. 
Regression analysis performed on the experience data presents 
the following relationship: 


Wee —. 192(X) + 1/248 R* = .04 
(OF) (2.06) 


The weak negative relationship indicates that as experience 


increases so does the agreement with this statement. Stronger 


Wes) 


support for this statement is coming from more experienced 
groups such as the prepare estimates work classification 
group. Notably there was a lack of consensus in the industry 
and corporate policy groups on the issue of DCAA operating a 
parametric cost estimating model for analysis purposes. 

The Dennedy article provides a description of a possible 
mechanism for accomplishing an agency running an offeror’s 
model. The article espouses having the offeror make the model 
available to the Government rather than providing a copy of 
the model to the Government as discussed in question 7a. The 
agency responsible for operating the model in the contractor 
facility is not addressed, however. The chief: advantage to 
this solution is the model remains a part of the estimating 
system that generated the offer in the proposal rather than a 
separate piece to be inserted into a negotiating position. 

S Summary 

Agreement was observed for all four of the question eight 
Statements. Minor areas of disagreement seem to be related to 
who would perform the work or function described rather than 
if the work should be performed at all. At most, 25% of the 
respondents chose to offer a recommendation for some 
organization other than DCAA to be tasked with the work 
described in any one of the statements. Additionally, less 


than 25% of the respondents disagreed with any one of the 
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statements in question eight. Written responses to these 
statements indicate a preference for two groups, DCMC and the 
buying organizations, taking on this function if DCAA does not 
gag. This desire to be involved in these aspects of the 
model evaluation could be beneficial. The team environment 
described by Dennedy, Deutsch and Hertling is a viable vehicle 
for combining the experience, knowledge, and needs of these 
groups to provide a single face to industry in another area of 
the acquisition process to address common goals. 
Dz TECHNICAL PROPOSAL EVALUATION IMPACTS 
Participants to question nine, shown below, were asked to 
record their opinions about the information that parametric 
methods could provide in technical proposal evaluation. 
Construction of cost estimating relationships used in 
parametric cost estimate based proposals should be used 
as a factor in evaluating a contractor’s level of 
understanding of the technical requirements in a 
seul Citation. 
Subsidiary question two and three issues are addressed in this 
statement. Source selection criteria and evaluation factors 
should take the entire proposal into account. Technreaw 
understanding and capability can be revealed in the way the 
offeror intends to manage the effort, the way the 


relationships in the cost estimate are related as well as in 


the verbiage in the technical proposal itself. 
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The average response value of 2.37 and median value of 2 
indicate a fair amount of agreement with this statement. 
Table 67 shows that the program manager and NASA groups were 
strongest in their agreement with the statement. Buying 
Organizations, U.S. Navy, support groups in the acquisition 
communbty, sand=contsuacting officers, particularly “Wore 


contracting officers, also agreed with this premise. 
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Analysis of variance of job description groups revealed 
that program managers, contracting officers, and auditors, 69 
respondents, agreed that CER construction should be considered 


as a technical evaluation factor. It also showed that the 
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technical support participants strongly disagreed with this 
statement although they do represent a very small group within 
the pool of respondents. This is an interesting response from 
those who would be likely to be assigned technical proposal 
evaluation duties. Table 68 contain the job classification 


analysis of variance data. 


Table 68 
Analysis of Variance For Question 9 Data 


DF 


Source: Developed by researcher. 





Weak negative relationships between responses and both 
GS/SES level and experience were also noted. The equations 


GS/SES Level 


Y = —-.336(X) + 7.29 Rea 13 
(ee ) (Ze, OM) 

Experience 

= 179K) +07.45 Rega 2 
(.09) (2.44) 


indicate that as GS/SES level or experience increases the 
agreement with the statement also increases slightly. This 
also supports the agreement levels seen in the auditor, cost 
estimating, and NASA groups. These groups are among the more 
experienced groups of all the demographic categories in the 


respondent pool. 
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Theme greemechias Wiikheeequestion mines» imdicates that 
consideration should be given to this aspect of technical 
proposal evaluation. With the strong disagreement from the 
technical support community, it may be prudent to involve 
someone from this community early in the process of developing 
an evaluation plan to encourage a team effort. Consideration 
should also be given during construction of the plan to 
allowing flexibility in the event both parametric-based and 
traditional cost proposals are submitted. As discussed in 
question 6g, fairness is essential in any source selection 
jolene’ < 
E. NEGOTIATIONS 

Question ten is a four part series of statements dealing 
with the effects that parametric cost estimating may have on 
the negotiation process. This process can be a factor in both 
pre-award as well as post-award circumstances as noted in 
question four. Issues in the fourth subsidiary research 
question are addressed in this series of statements. 

ai Cost or Price Negotiation Criticality 

The first area of interest in statement 10a, shown below, 
is the effect estimate accuracy might play in negotiations. 

Using parametric cost estimating methods should make 


negotiating costs less critical as cost estimate accuracy 
improves. 
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As estimates become more precise, will the issue of cost 
become less of an issue? The question assumes that cost will 
always remain an evaluation factor in any proposal evaluation. 
There was a lack of agreement and consensus among the 
participants on this issue. An average response value 
indicates disagreement but the median value of 2 indicates 
agreement with the statement. The program staff and industry 
cost analyst data displayed in Table 69 are the only 
demographic categories that exhibit any consensus on this 
ese . The program staff group iS unanimous in their 
disagreement that costs will become less important as estimate 
accuracy improves. The industry cost analysts see a shift in 
emphasis from costs to other factors as estimates become more 
precise. 


Table 69 
Question 10a Group Data 
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FAR 15.406-4 notes that cost will be included as a factor 
in source selection. This statement raises the question of 
how much will cost be a factor, if because of increasing 


accuracy, the acceptable range of both parties in a 
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negotiation is very narrow and overlapping. Parametric 
estimates derived from the same models and estimating systems 
for both offeror and buyer have the potential for this 
Situation if the offeror’s facilities are used as discussed in 
the statement 8d analysis. In the event that this occurs, the 
negotiations may shift from cost concerns to ones of a 
technical ability or schedule nature. Another possibility 
that is more likely is a shift in negotiations to inputs to 
the model and the assumptions that go with them if the model 
and estimating system are accepted as validated by both 
parties in a negotiation situation. 

2s Cost as an Independent Variable Impacts 

Statement 10b attempts to relate Cost As An Independent 
Variable (CAIV) with parametric cost estimating during the 
negotiation process. The second statement of question ten is 
shown as: 

Using parametric cost estimating methods should have 

little effect on negotiations as the concept of cost as 

an independent variable shifts emphasis to technical and 

schedule concerns for a given value. 
The concept of CAIV shifts emphasis to technical or other 
issues by making costs more of a constraint than an outcome of 
a series of events ina program. Prior to CAIV, there was an 
emphasis on cost control but it was still managed as a 


dependent variable in the program life cycle. CAIV introduced 
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the idea that costs would actually be controlled to some 
degree by working within some trade space with technical 
Capability, logistics supportability, schedule, and other 
concerns to produce and field affordable weapons systems in a 
timely manner. 

There was a large degree of diene with this 
statement. The majority of respondents to this question (104) 
either disagreed or strongly disagreed with this statement. 
The disagreement was somewhat of a consensus with an average 
value of 2.83, a standard deviation of .70 and a median value 
er, 3. Disagreement with this statement was widespread 
throughout the DoD acquisition community as shown by the group 
data in Table 70. Four of the largest groups of participants, 
Semeracting officers, "BGAA, DCMC, and the™U2S® Army had a 


consensus in their disagreement with this issue. 


Table 70 
Question 10b Group Data 
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The depth of feeling over this indicates that CAIV will have 
little effect during negotiations in shifting emphasis to 
other issues. 

oe Minor Impacts on Source Selection 

While cost realism may be a major part -of a source 
selection plan, it should be limited in its impact on other 
areas of an evaluation. All of the proposal component parts 
are interrelated to some degree but the factors should have 
some degree of independence. This statement explores the 
effects of parametric cost estimating methods and cost realism 
in source selection. 

Using parametric cost estimating methods’ should have 

little effect on source selection, they only affect cost 

realism. 
The aggregate data for this statement shows disagreement with 
an average response value of 2.66, a standard deviation of 
.78, and a median value of 3. Table 71 contains data from a 
major buying organization, USAF, and two vital components of 
the DoD acquisition community, the auditor and cost estimator 
groups, that disagree with this statement. 

This is contrasted with the regression analysis performed 
with the business size data. The equation and data 


Y =F coe 7 Se Ro =q@l's 
Cale?) (2.0) 
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display a negative relationship between business size and the 
response value for this statement. As the business size 
increases so does the level of agreement with this statement. 
The industry and corporate policy groups do not show any 


consensus in their opinion on this statement, however. 


Table 71 
Question 10c Group Data 
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4. Significant Impacts on Source Selection 

The interrelated nature of the parts of an offeror’s 
proposal could make the effects of one element have a 
disproportionate effect in the source selection process. 
There was a similar lack of consensus on statement 10d (shown 
below) as there was in statement 10c. 

Using parametric cost estimating methods should have a 

major effect on source selection, they affect past 

performance, technical, management, and cost factors in 

proposal evaluation schemes. 


An average response value of 2.48 and a median value of 2 was 


observed. The distribution of responses was strongly agree, 
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20, agree, 6/, disagree, 68, and strongly disagree 16. 
Relatively few people (39) chose no opinion or did not respond 
to this statement. The DCAA and corporate policy group 
responses in Table 72 are indicative of the range of opinion 
to this statement. They are not, however, indicative of the 
degree of consensus on this issue. These two groups were the 
only ones to show any sort of consensus of the demographic 
groups analyzed. The contrasting opinions between industry 
and DCAA in the acquisition community is clear. The industry 
business size regression analysis results 


Y Sa. 3caieee- 2.77 R* = .22 
(26) Zr. 1) 


show a positive relationship between the responses and the 
size of the business. The smaller businesses strongly agree 
with the statement while the larger ones merely agree. 


Table 72 
Question 10d Group Data 
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are consistent. The regression analysis demonstrates opposite 
trends for these two contrasting statements, and from this one 


cOouldm@einterpret Pitat “the “industry “portion of the Dee 
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acquisition community views a proposal as a highly integrated 
whole rather than separate sections. The Government, however, 
may not see the issue in the same way. DCAA responding to 
this statement and the technical support group in responding 
to question nine disagree with the idea that using parametric 
cost estimating methods can have significant effects on the 
source selection and subsequent negotiation processes. 

Dennedy, Deutsch, and Hertling also disagree with the 
premise that the use of parametric cost estimating methods can 
have an effect on negotiations. [Ref 8:p.3] The negotiation 
process 1S based on estimates of all types and parametric 
estimates are only one of many sources of information. in 
total, the results of statements 10a through 10d indicate that 
the DoD acquisition community does not believe that the use of 
parametric cost estimating methods will change the fundamental 
nature of negotiations. It will simply redefine some of the 
points around which the negotiations may flow. 

SUMMARY 

This chapter and the preceding two chapters described the 
data collection efforts and analyses conducted for this 
thesis. Data collection was chiefly done through a ten 
question survey distributed to members of the DoD acquisition 
community. Personal interview information was also collected 


and presented with each gquestion’s survey information as it 
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pertained those questions or statements. The combined data 
were examined for indications regarding the overall opinion of 
the entire DoD acquisition community about each issue. The 
demographic group data were then examined and contrasted with 
the entire group responses and between group responses. Any 
groups within the database that had any opinion that was 
markedly different from that of the entire respondent pool 
were highlighted. An analysis of this information was 
presented for each segment of each question. The final 
chapter in this thesis will conclude the study with 


conclusions and recommendations made from the analysis. 
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IX. PRINCIPAL FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 


A. PURPOSE 

The primary purpose of this study was to examine the 
effects of increasing the use of parametric cost estimation 
methods in individual contract actions. This chapter will 
provide conclusions about those effects and recommendations 
for expanding the part that parametric cost estimating methods 
play in Federal Government acquisition processes. Principal 
findings and conclusions will be addressed and then they will 
be followed by the recommendations. Areas of further research 
will be the final section of this thesis. 
Br CONCLUSIONS 

The DoD acquisition community views the use of parametric 
cost estimating as having limited applicability. Areas that 
can be termed traditional uses have the highest degree of 
utility in questions one through four. The acquisition 
community believes that the methods are best applied to 
developmental applications and serve little purpose outside 
those areas. The data support the conclusion that the DoD 
acquisition community is not familiar with the techniques and 
methods used in parametric cost estimating. They do not have 


an appreciation of the benefits that parametric cost 
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estimating can provide across the entire spectrum of activity 
in the DoD acquisition process. 

Practical experience with using the methods in the 
acquisition process and research on the possible applications 
for the methods provide a different view. NASA, industry, and 
the control group responses clearly showed aes the methods 
can be used successfully in a far wider range of activities 
than they are normally applied by DoD agencies. They also 
demonstrated areas that have clearly not benefitted by the use 
of parametric cost estimating methods, such as contracting for 
commodities, T&M/Labor hour contract situations, and in most 
areas of the PF/D&OS phase. Parametric cost estimating also 
may be applied in those areas, however, currently they are not 
being used there. The methods may not provide enough 
improvement over existing estimating systems to make 
investments in them cost effective. 

The potential effects of using parametric cost estimating 
methods will require some changes to be made to the way data 
are collected, reviewed, presented, and analyzed. However, 
they will not fundamentally change the acquisition process. 
The DoD acquisition community views parametric cost estimating 
methods as an incremental step in increasing the efficiency of 
the acquisition process. The community acknowledges the fact 


that the methods can affect almost every action in the DoD 
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procurement process. They also note that if the methods are 
to be used, they must be considered from the initial planning 
Stages. 

The members of the acquisition community also are viewing 
the use of parametric cost estimating methods- in terms of 
acquisition reform. The data demonstrate a preference for 
giving the contractor community latitude to use the methods as 
they fit best in the contractor’s systems rather than as 
requirements to be met. The data also indicate that the 
acquisition community would accept the idea that teaming 
concepts have potential benefits to parametric cost 
estimating. Major stakeholders in the community identified 
each other as important members of the process whereby a 
parametric cost estimating model is calibrated and validated 
for use. The use of teaming and parametric cost estimating 
fit in well with the concept of disengagement by DoD in its 
contractor estimation system review programs. 

CA RECOMMENDATIONS 

The primary recommendation of this thesis is to maintain 
the Parametric Cost Estimating Initiative and the Joint 
Industry/Government Working Group. This initiative, through 
the working group, has been instrumental in bringing awareness 
to the benefits that parametric cost estimating can provide. 


The workshops which the working group sponsors and newsletters 


1o3 


tiie wOMindeEshes aneyoubstandingsfiora for publicizingsethese 
benefits, exploring new areas in which the methods can be 
applied, and producing meaningful discussion about actual 
application of the methods in specific situations. The large 
number of surveys not answered indicates that the majority of 
the DoD acquisition community is still unfamiliar with 
parametric cost estimating methods. 

A second recommendation is to evaluate and publish 
results from the Joint Government/Industry Parametric Cost 
Estimating Pilot Program. The lessons learned and successes 
will help to promote the use of the methods and also to 
publicize how they can benefit both Government and Industry. 
Business case analyses could be performed with these results 
to quantify the effects of using parametric cost estimating 
methods and implementation procedures. 

Training the .acquasition™ community 1s the  thatag 
recommendation of this thesis. One of the major efforts of 
the working group has been to establish a training program in 
the area of parametric cost estimating. The training should 
be made readily available to the acquisition community as 
rapidly as possible to reap the maximum benefits from the use 
of the methods. Training should also include a review of the 
existing curricula for acquisition workforce professionals. 


A brief discussion regarding parametric cost estimating 
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methods to introduce them to the workforce could provide basic 
awareness and understanding of the methods if it were added to 
existing cost and pricing course materials. 

Parametric cost estimating methods should be considered 
for incorporation in DoD modeling and simulation: programs for 
acquisition purposes. The U.S. Navy Acquisition Center for 
Excellence (ACE) could potentially benefit from the use of the 
methods as well as provide awareness to the acguisition 
community about their potential applications. The U.S. Navy 
Research Development and Acquisition Team Strategic Plan for 
1996 and 1997 provides several other areas where the methods 
could be applied. The strategic goals for total ownership 
cost and innovation/technology insertion discuss modeling and 
Simulation areas that could also benefit from using parametric 
cost estimating methods. 

Connectivity between databases should be improved. As 
more of the DoD systems and equipments incorporate COTS and 
NDI, integration will become a major cost driver. Parametric 
estimating could be helpful in modeling the integration 
schedule and costs if databases for the individual equipments 
could be integrated. Parametric estimation of total life 
cycle costs could also benefit from greater connectivity 


between databases. 
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D. FURTHER RESEARCH TOPICS 

The following is a list of several areas that might be of 
interest and benefit to the acquisition community regarding 
parametric cost estimating: 


1. Business case analyses of the Government agencies and 
companies involved in the Joint Government/Industry 
Parametric Cost Estimating Pilot Program could provide 
greater insights to the application of parametric cost 
estimating methods to DoD acquisition. 

2. A review of how the use of parametric cost estimating 
methods affect the DoD estimating system review program 
would be useful to buying organizations as well as DoD 
administrative agencies. 

3. Business case analyses of parametric estimating 
applications to the concept of total life cycle costs 
would provide benefits to future programs and DoD 
managers. 

4. A review of the regulatory and other barriers to 
effective implementation of using parametric costing 
could benefit those interested in promoting the wider use 
of the methods. 


E. FINAL SUMMARY 

The final summary of the issues addressed in this thesis 
will be a restatement of the research questions followed by 
brief answers. 
Primary Research Question: 


How could a DoD emphasis on using parametric methods in 


cost estimating affect the procurement process? 
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Answer: A DoD emphasis on parametric methods in cost 
estimating could influence almost every step in the 
acquisition process in some way. The effects that they 
would cause would be minor in nature, however. 
Subsidiary Research Questions: 
1. What changes could offerors make to their proposal 
preparation, submission, and support processes. to 
accommodate using parametric methods, techniques, or 
software in estimating potential contract costs? 
Answer: Offerors desiring to incorporate parametric cost 
estimating methods into their proposal preparation 
processes could either use a commercially available model 
or develop one of their own. In either case a database 
from which to work is essential and must be constructed. 
The decisions made here will influence the ability of the 
offeror to respond successfully to Federal Government 
Selacitabions. 
2. What impacts could emphasis on using cost estimates 
based on parametric estimating techniques have on source 
selection criteria and on the evaluation of proposals by 
DOD? 
Answer: Cost estimates based on parametric estimates 
could impact the entire source selection process. They 


have impacts on the evaluation plan, the proposals 
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submitted, and thus the material available for evaluation 
and source selection decisions. 

3. How could source selection criteria and evaluation 
factors be constructed to fairly evaluate a proposal that 
uses a parametric cost estimate with one that does not? 
Answer: Source selection criteria must be flexible enough 
to include proposals based on parametric estimates as 
well as ones based on traditional estimates. Failing to 
do so could impact the level of competition in a given 
procurement. 

4. What effects could using parametric cost estimations 
have on negotiations? 

Answer: Parametric cost estimating methods should have 
little effect on negotiations. The points around which 
one negotiates may shift to inputs to the model rather 
than specific cost elements but the basics of negotiation 


will remain the same. Cost will always be a factor. 
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APPENDIX A. DEFINITIONS 


Analogy Model - a model that estimates costs of a new program 
or system from data on past costs of similar programs or 


systems. [Ref 23:p.290] 


Analysis of Variance - an expression of the measure of total 
variation in a data set as a sum of terms, which can be 
attributed to specific sources, or causes of variation. [Ref 


Seo. 2 07 | 


Benchmarking Process Costs - generating an estimate of the 
costs associated with a process for later comparison with 


other estimates or actual costs. 


Bottoms Up Cost Estimate (Detailed Cost Estimate) - an 
estimate of project component costs from detailed 
understanding of the cost elements of each component. 
Supplier quotations, labor hour estimates from time and motion 
studies or labor hour and material estimates from learning 
curve analyses form the basis of these estimates. [Ref 


2:p.294] 
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Calibration - the process of indexing a parametric model or 


System eo isgical Cost and product history. [Ref 5:p.4] 


Capacity Cost Estimates - estimates based on logarithmic plots 
of similar plants which show a relationship between capacity 


and costs. [Refei4:p2132Z] 


Coefficient of Determination - the proportion of variation in 
the dependent variable that has been explained or accounted 


for by the regression line. [Ref 37:p.46] 


Commercial~Off-The-Shelf - an existing system-or equipment 
that is sold to the Federal Government in the same 
configuration that it is sold to the general public or with 


MinoOnpeMmediGrcatvons. [Ref Sep.42] 


Cost Analysis - the review and evaluation of a contractor’s 
costs or pricing data, and of the judgmental factors applied 
in projecting from the data to the estimated costs, for the 
purpose of determining the degree to which the contractor’s 
proposed costs represent what contract performance should 


cost, assuming reasonable economy and efficiency. [Ref 3:p 19] 
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Gast or PricingsData —“all facts™e@har.. Drudene buyers oma 
sellers would reasonably expect to affect price negotiations 


segmaticantly. (Refi 13:part, 15.3801} 


Cost Realism - means the costs in an offeror's: proposal are 
realistic for the work to be performed, reflect a clear 
understanding of the requirements and are consistent with the 
various elements of the offeror's technical proposed. [Ref 


fereart 15.801] 


Dependent Variable - the variable whose value is to be 


weedicted. [Ref 37:p.39] 


Eguipment-Ratio Estimates - estimates derived from process 
flow diagrams and eguipment lists with installation and 
construction costs estimated as a ratio of past similar 


efforts. [Ref 14:p.132] 


Expert Opinion - the use of the judgment of experts when 


Supporting data or program definitions are insufficient. [Ref 


23:p.290] 


Independent Variable - the variable about which knowledge can 


mewobtained. [Ref 37:p.39] 
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Layout Estimates - estimates from plant layouts and process 


flows and equipment lists. [Ref 14:p.132] 


Learning Curve - an empirical relationship between the number 
of units of an item produced and the number of units of a 
resource used to produce them. The most common relationship 
used 1s units of production and number of labor hours required 


COpproduceg the unies, [Ref Psp l7267] 


Negotiation - a process between buyers and sellers seeking to 
reach mutual agreement on a matter of common concern through 


fact finding, bargaining, and persuasion. [Ref 3:p.41] 


Non-developmental Item - an item developed exclusively at 
private expense and sold in substantial quantities, on a 
competitive basis, to multiple state and local governments. 


[Ret lStpart 2.1 0del 


Parametric Cost Estimate - an estimate derived from 
statistical correlation of historic system costs with 
performance and/or physical attributes of the system. [Ref 


Si akg. 
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Performance Specification - a description of a deliverable in 


terms of desired operational capabilities. [Ref 3:p.44] 


Preliminary Bill Estimates - estimates derived from bills of 
material produced from initial drawings and labor-material 


PeeltOos from past worksae[Ref 174:p.132) 


Price Analysis - the process of examining and evaluating a 
proposed price without evaluating its separate cost elements 


aaaeproposed profit. [Ref 1i3:part 15% 8] 


Regression Analysis - the mathematical nature of the 


association between two variables. [Ref 37:p.39] 


Risk - the probability of not attaining the goals for which 


the party entered a contract. [Ref 3:p.51] 


Source Selection - the process wherein the requirements, 
facts, recommendations and policies relevant to an award 
decision in a competitive procurement of a system/project are 


eXamined and the decision is made. [Ref 3:p.53] 


205 


Standard Error of the Estimate - the standard deviation of the 
population of y values predicted from a single value of xX. 


[REE 23352305 | 


Top Down Cost Estimate - an overall cost estimate for a 
proposed project based on the global properties of the project 
from which component parts are partitioned for planning 


purposes. [Ref 34:p.176] 


Validation - the process or act of demonstrating the 
calibrated model’s ability to function as a credible forward 


estimating tool or system. [Ref 5:p.4] 


Work Breakdown Structure - a product-oriented family tree 


resulting from system engineering efforts which completely 


define the program. [Ref 37:p.176] 
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APPENDIX B. LIST OF ACRONYMS 


sd? - Sum of the Squares 


6 — Population Variance 


ACE - Acquisition Center of Excellence 


ACO - Administrative Contracting Officer 


ACT —- Active Coefficient 


eG - 


aly - 


Cost Analysis Improvement Group 


Cost As An Independent Variable 


> — Cost Accounting Standards 


ee — Concept Exploration 


eer — Cost Estimating Relationship 


eecA -— 


COCOMO 


eels — 


Sear — 


eek — 


eeu - 


eekG — 


DCAA - 


aeaiC 


Pvc — 


Sompetlithen in Contracting Act 

- Constructive Cost Model 
Commercial off the Shelf 
Cost-Plus-Award-Fee Contract 
Cost-Plus-Fixed-Fee Contract 
Cost-Plus-Incentive-Fee Contract 
Contract Pricing Reference Guides 
Defense Contract Audit Agency 
Defense Contract Management Command 


Development Cost Variable 


DF - Degrees of Freedom 


DFARS - Defense Federal Acquisition Regulation Supplement 
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DEStE ey sbetense  Logistiucsesiudrves, Informations Exchange 
DoD - Department of Defense 

DoE - Department of Energy 

DON - Department of Navy 

DTIC - Defense Technical Information Center 

ECP - Engineering Change Proposal 

E&MD - Engineering and Manufacturing Development 

FE Seti esigat is tue 

FAR - Federal Acquisition Regulation 
hheS=sthomerixed-Paice Goneract 

FPE - Fixed-Price with Economic Price Adjustment Contract 
FPIF - Fixed-Price Incentive Firm Contract 

FPIS - Fixed-Price Incentive, Successive Targets Contract 


FPRA - Forward Pricing Rate Agreement 


FPR(P) - Fixed-Price with Prospective Redetermination Contract 
FPR(R) - Fixed-Ceiling-Price with Retroactive Redetermination 
COneEnRACE 


ICE - Independent Cost Estimate 

GS - General Service 

GSA - General Services Administration 

IPCE - Independent Parametric Cost Estimate 
LG - Large Ship Factor 

LRIP - Low Rate Initial Production 


LSBF - Least Squares Best Fit 
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NASA - National Aeronautics and Space Administration 

NDI - Non-developmental Item 

OSD - Office of the Secretary of Defense 

P - P value 

feen= Procuring Contracting Officer 

POeRR — Program Definition and Risk Reduction 

PF/DOS - Production, Fielding/Deployment and Operational Support 

PRICE - Programmed Review of Information for Costing and 
Evaluation 

PROC - Procurement Cost Variable 

R* - Coefficient of Determination 

RDT&E - Research, Development, Test, and Evaluation 

SES - Senior Executive Service 

ee —somall Ship Factor 


SOW 


Statement of Work 

SPI - Single Process Initiative 
fous otandard Sum of the Errors 
T&M - Time and Materials Contract 
TINA - Truth in Negotiations Act 
USAF - United States Air Force 
USA - United States Army 

USN - United States Navy 


WBS - Work Breakdown Structure 
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APPENDIX C. SURVEY QUESTIONS 


Please use the following as a definition, taken from the DoD 
Parametric Cost Estimating Handbook published in the fall of 
HOE9>, Lor Parametric Cost Esti maskamncquetBCe) : 


“A technique employing one or more Cost Estimating 
Relationships (CERs) and associated mathematical relationships 
and logic. The technigue is used to measure and/or estimate 
the cost associated with the development, manufacture, or 
modification of a specified end item. The measurement is 
based on the technical, physical, or other end item 
characteristics.” 
PARAMETRIC COST ESTIMATING SURVEY 
Please complete the following demographic information. 
1. What is your job classification? (Select 1 that fits best or write in a specific title 1f none apply.) 


Cost Estimator Cost/Price Analyst Contract Negotiator 
____ Contracting Officer Program Staff Program Manager Engineer 
Legal Support Technical Support Auditing Accounting 
Other 


2. If employed by industry, annually does your company do business with the Federal 
Government totaling: 
< $100 Million $100 - 500 Million $500-$1 Billion >$1 Billion 


3. If employed by the Federal Government: 
USA DCMC DCAA NASA USN USAF GSA 
Other: GS/SES Level/Rank 


4. How long have you been working in the cost estimating or acquisition fields? 
(Please select either career field and the appropriate length of time.) 
___One year or less 1-5 years 5-10 years 10-20 years ___— Over 20 years 


5. What functions involving cost estimating do you primarily perform? 
Prepare Estimates Use in Negotiation Preparation or Proposal Analysis 
Review/Audit Estimates Set Corporate or Federal Government Policy 





6. What is your level of mathematics knowledge? 

____ High School Algebra/Calculus ____ College Algebra __— College Business Calculus 
____ College Engineering Calculus Senes ___ College Mathematics Minor/Major 

____ Advanced Degree Involving Training in Mathematics Other 
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Please record your opinions regarding the following questions relating to PCE. 
For questions | through 4, use the following scale to represent the utility of PCE in the listed 
areas. 

(1) Vital (2) Very Useful (3) Somewhat Useful (4) No Use (5) Hinders the Process 

1. How useful is PCE in these areas? 

















Bench Marking Process Costs Validating Bottom’s Up Estimates 
Program Estimates ___ Budgeting ICE Generation Software Contracts 
Repair Parts Contracts ____ R&D Contracts Service Contracts 

Hardware Contracts ___ Construction Contracts Commodity Contracts 


PCE is also useful for: (Please fill in other areas.) 


2. How useful is PCE in each of the stages of a system’s life cycle? 

Concept Program Definition Engineering & Manufacturing Production, Fielding/ 

Exploration & Risk Reduction Development Deployment & Operational 
Support 


3. How useful is PCE in situations where you would also use the following contract type: 
FFP FPE FPIS FPR(P) FPR(R) FPIF CPIF CPAF CPFF T&M _ Labor Hour 





PCE 1s also useful for this contract type situation . (Please fill in others.) 


4. How useful is PCE in estimating costs for activities in these two. areas of contract 
management? 
Pre-Award Actions Contract Post-Award Actions 


This part is a series of statements about the use of PCE methods. Using the following scale, 
please indicate to the left of each statement, the degree to which you agree or disagree with 
them. 

(1) Strongly Agree (2) Agree (3)Disagree (4) Strongly Disagree (5) No Opinion 


5. PCE methods, models and software: 


a. Could eventually replace traditional “bottoms up” cost estimating methods in many 
acquisition estimating applications as a result of acquisition streamlining. 

b._____ Could replace traditional cost estimating methods as larger, more compatible data bases, 
greater computing power and improved software become available. 

c: Can save contractors and Federal Government oversight officials significant time and 
expense in developing and then evaluating proposals. 

d. Could significantly change the way potential contractors collect and summarize cost 
information. 

er Should decrease the probability of cost overruns in programs where used. 

f.___ Should change the way Cost and Pricing Data submitted to support a proposal is viewed. 


6. In solicitations, buying activities should: 


a.____ Encourage that PCE be used for specified procurement actions. 

b.____ Specify PCE methods, models, or software for cost proposal construction. 
c.._____ Leave PCE methods, models, or software selection to contractor discretion. 
d.____ Require a specific cost breakdown format for PCE based cost proposals. 


Ze 


= Require a specific format to extract labor hour, subcontractor, or maternal quantity 
estimates for technical proposal evaluation. 


fi State criteria to be used to evaluate the statistical significance of CERs used to support 
a PCE model. 
g. Require cost proposals based on either PCE methods or traditional costing methods. 


Evaluation schemes cannot fairly distinguish between proposals using different costing 
methods for the same work. 


7. A potential contractor should provide the following to support a cost proposal: 


a. PCE model software and input data used. 
b. Justification to support the use of PCE models and input data. 
ce. Data and rationale used to validate a PCE model for a specific set of circumstances. 


8. DCAA should perform the following activities: (If not DCAA, who should do them?) 
a._____ Establish metncs to determine a PCE model’s validity in a given situation. 

b.____ Establish cntena to determine when a PCE model should be updated. 
c..__—~Vernify that a PCE model accurately predicts costs for the expected contract work. 
ds “Run” a contractor’s PCE model for purposes of analyzing the cost proposal. ___ 


9. Construction of CERs used in PCE based proposals should be used as a factor in 
evaluating a potential contractor’s level of understanding of the technical requirements in a 
solicitation. 


10. Using PCE methods: 


a. Should make negotiating costs less cntical as cost estimate accuracy improves. 

b. Should have little effect on negotiations as the concept of Cost as an Independent 
Variable shifts emphasis to technical and schedule concerns for a given value. 

c.___ Should have little affect on source selection, they only affect a proposal’s cost realism. 

d.____—sdShould have a major affect on source selection, they affect past performance, technical, 


management, and cost factors in proposal evaluation schemes. 


213 





APPENDIX D. INTERVIEW QUESTIONNAIRE 


.How should an emphasis on uSing parametric methods in cost 
estimating affect the procurement process? 


.How could the use of parametric cost estimates be incorporated 
into program management beyond their traditional uses? 


.How far should DoD/DON go in emphasizing parametric cost 
estimating methods? 


.Should the use of parametric cost estimates be required for 
certain procurement actions? 


.What information should DoD/DON agencies expect potential 
contractors to provide in support of their proposals that use 
parametric cost estimating techniques iene proposal 
preparation? 


.DOoO you see any major changes to the way solicitations are 
structured, proposals are generated, proposals evaluated, and 
sources selected because of an emphaSis on uSing parametric 
cost estimating techniques? 


.What support should DoD/DON organizations expect of DCAA and 
DcMC for analyzing parametric cost estimate based proposals? 


.Should DoD/DoN provide guidance on how the variance between a 


parametric cost estimating model and the work described on a 
solicitation be measured? 


p23) 
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APPENDIX E. COLLECTED DATA 


DEMOGRAPHIC GROUP INFORMATION 


Group Name Sort | #of Math | Math | Experience | Experience 
Code | Responses/ | Sort Sort | Sort Code | Sort Code 
%Yof Code | Code | yu O 
Responses LL 


O E 
ice | iu sx| | | (= 





cman) | |) 
Progam Manse [6 [739% [ose [uw [are [oo 
—_ i. > | | Po. 
eciclSuppon [> [5 _20% [see [120 [2a3 
me | fom! | | | 
= ee ee 
ioowion [a |2 we fovo [io aso 
sso0Miin-siiion [we [1 swfovo | | | 
site [is fo asm fovo [rw [ass [oe 
a. | «lee of 
Se Te 


WAI ji 


Group Name Sort | #of Experience 
Code | Responses Sort Code 


Government — <_ oe a dei 
rr, 
DCA 0 
U.S. Navy 22 10.6% 73 Aas — 





QO} 
7 


ieee 


Total Government ae 189 90% 
Level/Rank a ar 

GS 12/03 = 

re aw 

iad Pe" 


GS 15/06 ys 1.5% 


102 Fee mato 28. 


er 


Experience 


Cost Estimating 
Community 


23 
Acquisition Community 24 





One Year or Less 


One to Five Years 


Five to Ten Years =a 19 9.1% 1.54 
28 |85 40.5% 1.57 
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Ten to Twenty Years 





Group Name Sort | #of Math | Math | Experience | Experience 
Code | Responses Sort Sort Sort Code 
Code | Code 
UL O 


Dinars) | | | 

37.4 
33 | 24% | 39. 

Set Government Policy 34 [28 13.3% 38.2 

Mathematics Knowledge Los | i 2 ee 
36 


High School 24 11.4% 28.3 55 
Algebra/Calculus 


Set Corporate Policy 


64 30.5% 


College Algebra 
College Business Calculus | 37 69 32.9% 
34 


College Engineering 16.2% 
Calculus 

College Mathematics 31 14.8% 
Major or Minor 


S18) 
34 
35 
36 
38 
39 
Advanced Degree With 40 28 13.3% 
Mathematics 


= ” 
ae 


SURVEY DISTRIBUTION AND RETURN DATA 


# Incorrect | # Surveys 
Addresses | Returned 





# Incorrect | # Surveys 
Addresses | Returned 
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TOTAL GROUP DATA 
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Avg | Std Var 
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[encome dn [x [= [= [» [2 | [ew fiw [= [ow fos | 
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SS ee eee EEE 
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ee Ce Ee ee ee Eee 
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Cost Estimator Group Data 
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Cost/Pnce Analyst Group Data 
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Negotiation Preparation Group Data 
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Review or Audit Estimates Group Data 
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Set Corporate Policy Group Data 
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Federal Government Policy Group Data 
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